3 Commits

Author	SHA1	Message	Date
YiMing Han	bc7cb7d0e8	Revert "only dart" ... This reverts commit 01df3731d6.	2023-08-18 15:13:59 -04:00
YiMing Han	01df3731d6	only dart	2023-08-18 15:09:24 -04:00
Andrej Karpathy	4c6f0af9ff	add the ability to train a custom sentencepiece tokenizer with a given vocab_size, and pretok with it. some more changes still needed to merge this branch, in train.py and ofc run.c. did this in a sadly bit ugly, but fully backwards compatible way. basically when we use custom tokenizer we create a whole new directory structure for that	2023-08-11 03:58:22 +00:00