3 Commits

Author	SHA1	Message	Date
Aniket	32cecbfe4a	freeing tokenizer in test.c	2023-08-26 16:35:50 -04:00
Andrej Karpathy	4b3e66021a	lol text	2023-08-23 00:26:47 +00:00
Andrej Karpathy	d1eb18b8ec	add BOS and EOS function to the Tokenizer as we start to converge closer to the Llama 2 code from Meta, and as we're about to add the Chat capability	2023-08-23 00:08:22 +00:00