3 Commits

Author	SHA1	Message	Date
atamyrat	c02865df30	prompt tokenizer improvements: utf8 support, add_dummy_prefix and byte_fallback options to match sentencepiece	2023-08-07 13:12:44 +03:00
Andrej Karpathy	b4bb47bb7b	big change: adding prompting. many LOC, but critical. ty @atamurad for the first draft, i ended up tuning it quite a bit.	2023-07-28 04:12:54 +00:00
Andrej Karpathy	3bfa5665d1	delete the run_wrap file! yay. ty @python273 and @ggerganov for code snippets	2023-07-24 04:02:57 +00:00