prompt tokenizer improvements: utf8 support, add_dummy_prefix and byte_fallback options to match sentencepiece

Browse Source

...

This commit is contained in:

atamyrat

2023-08-04 04:18:20 +03:00

parent 3c3b19b14c

commit c02865df30

3 changed files with 38 additions and 10 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

tokenizer.bin

BIN

Binary file not shown.