bigchange: add multiquery support in run.c. we can now train and inference multiquery models (where n_kv_heads < n_heads). this also means that we, in principle, support Llama 2 34B and 70B models, which are multiquery

2023-08-13 19:34:05 +00:00
parent 9ff459b925
commit 38bfac90a8
5 changed files with 33 additions and 26 deletions
@@ -294,7 +294,6 @@ If your candidate PRs have elements of these it doesn't mean they won't get merg

 - revive tests; train a tiny Llama test model (committed to repo) and use it as reference in unit tests
 - make it easier to add a new dataset with not too much pain
- add multiquery support into run.c
 - should calculate freq_cis online in the script run.c instead of loading them
 - int4/8 quantization
 - export the model in a more sensible output format with a proper header, etc.