## Tokenizers in src/mistral_common/data ##

The older sentencepiece tokenizers in src/mistral_common/data
are in protobuf format:

mistral_instruct_tokenizer_240216.model.v2
mistral_instruct_tokenizer_240323.model.v3
mistral_instruct_tokenizer_241114.model.v7
mistral_instruct_tokenizer_241114.model.v7m1
tokenizer.model.v1

While not strictly speaking text files, they are mostly human readable
and could be patched via a transformation to/from text using sentencepiece.

Note: The more recent Tekken tokenizers are defined in json.
