Tokenizer Arena
Easily compare between different tokenizers simultaneously
Create by
Adithya S K
on top of
Transformer js
gpt-4o
gpt-4 / gpt-3.5-turbo / text-embedding-ada-002
text-davinci-003 / text-davinci-002
gpt-3
Grok-1
Claude
Mistral
Gemma
LLaMA / Llama 2
Llama 3
Phi 3
Cohere Command-R
T5
bert-base-cased
Custom Tokeniser from HF
Tokenizer Name: Xenova/gpt-4o
Tokens: 0
Characters : 0
Remove
Tokenizer Name: Xenova/gpt-4
Tokens: 0
Characters : 0
Remove
Text
Token IDs
Hide
Show Visualization