bigchange: add multiquery support in run.c. we can now train and inference multiquery models (where n_kv_heads < n_heads). this also means that we, in principle, support Llama 2 34B and 70B models, which are multiquery

2023-08-13 19:34:05 +00:00
parent 9ff459b925
commit 38bfac90a8
5 changed files with 33 additions and 26 deletions
@@ -53,7 +53,6 @@ if compile:
    model = torch.compile(model) # requires PyTorch 2.0 (optional)

 # load the tokenizer
-assert checkpoint["config"]["dataset"] == "tinystories" # TODO: generalize
 tokenizer_model = get_tokenizer_model_path(vocab_size=gptconf.vocab_size)
 enc = Tokenizer(tokenizer_model=tokenizer_model)