toksuite/supertoken_models-llama_tiktoken-gpt-4o
Text Generation
•
2B
•
Updated
•
57
Edit Models filters
Apps
Inference Providers
Active filters:
bpe
nursimakgul/turkish-fen-tokenizer-v1
Updated
toksuite/supertoken_models-llama_CohereLabs-aya-expanse-8b
Text Generation
•
2B
•
Updated
•
54
Rogarcia18/wikitext2-bpe-tokenizer
Updated
PABITRA07/hindi-english-BPE-nmt
Updated
Msok99/lfm2-khmer-merged-18k
Updated
SherifAnar/russian-bpe-16k
Updated
mjbommar/glaurung-binary-tokenizer-001
Feature Extraction
•
Updated
mjbommar/glaurung-binary-tokenizer-002
Feature Extraction
•
Updated
•
1
pandurangpatil/sample-marathi-bpe-tokenizer
Updated
OpenMachineAI/UnitronX-Tokenizer-32k-v1
Updated
ansul90/hindi-bpe-tokenizer
Updated
vnaren13/telugu-bpe-tokenizer
Updated
agileabhi/hindi-bpe-tokenizer
Token Classification
•
Updated
dar5115/NLP_BPE
Updated
mjbommar/binary-tokenizer-001-4k
Feature Extraction
•
Updated
mjbommar/binary-tokenizer-001-8k
Feature Extraction
•
Updated
mjbommar/binary-tokenizer-001-16k
Feature Extraction
•
Updated
mjbommar/binary-tokenizer-001-64k
Feature Extraction
•
Updated
mjbommar/binary-tokenizer-001-32k
Feature Extraction
•
Updated
shwethd/kannada-tokenizer-50k
Updated
Soldier-Boy/twitch-tokenizer
Updated
alea-institute/kl3m-multi-word-001-4k
Fill-Mask
•
Updated
alea-institute/kl3m-multi-word-001-8k
Fill-Mask
•
Updated
alea-institute/kl3m-multi-word-001-16k
Fill-Mask
•
Updated
alea-institute/kl3m-multi-word-001-32k
Fill-Mask
•
Updated
alea-institute/kl3m-multi-word-001-64k
Fill-Mask
•
Updated
alea-institute/kl3m-multi-word-001-128k
Fill-Mask
•
Updated
arabovs-ai-lab/TatarTokenizers
Updated
alea-institute/kl3m-multi-word-002-4k
Fill-Mask
•
Updated