Skip to content

v1.0.1

Compare
Choose a tag to compare
@Hk669 Hk669 released this 27 May 17:28
· 26 commits to main since this release

first release

adds the following functionalities:

  • BPETokenizer: which can be used to build your tokenizer for the LLM
  • Tokenizer: a base class which leverages the save and load of the vocab and merges