Tools mentioned in this article
Open the browser-based tool while you read and try the workflow immediately.
Building on LLM APIs, you constantly need to know “how many tokens is this prompt?” and “does this fit in the context window?” The LLM Token Counter answers both: paste text and see token counts for Claude, GPT and Gemini side by side — entirely in your browser.
What is a token?
A token is the smallest unit an LLM processes. Models split text neither by characters nor by words, but into subword units via a tokenizer.
English unbelievable, for instance, may split into fragments like un / believ / able. For English text, roughly 4 characters ≈ 1 token is a good rule of thumb.
Token counts matter because two things are priced in them:
- API cost — both input and output are billed per token
- Context window — the model’s limit on how much text it can handle at once (e.g. 200K tokens)
Every model family tokenizes differently
The same text produces different token counts on different models.
| Model | Tokenizer | Availability |
|---|---|---|
| GPT-5.5 / GPT-5 / GPT-4o / GPT-4.1 | o200k_base | Open source (tiktoken) |
| GPT-4 Turbo / GPT-3.5 | cl100k_base | Open source (tiktoken) |
| Claude 3 and later | Proprietary | Not public |
| Gemini | SentencePiece-based | Not public (count API only) |
OpenAI publishes its tokenizers as open-source tiktoken, so GPT tokens can be counted exactly in browser JavaScript. Anthropic has not published the Claude 3+ tokenizer, and Google provides no client-side tokenizer either.
This tool therefore counts GPT exactly with o200k_base, and estimates Claude and Gemini as “the exact GPT count × a correction factor based on the text’s character mix”. Estimates can be off by roughly ±10–20%, which is why the UI labels them clearly as approximate.
Non-English text costs more tokens
Tokenizer vocabularies are trained predominantly on English, so frequent English words compress into single tokens, while other scripts fragment. The effect is dramatic for CJK text:
The quick brown fox jumps over the lazy dog. (44 characters) is 10 tokens in o200k_base. The Japanese sentence 吾輩は猫である。名前はまだ無い。 (16 characters) is 14 tokens — roughly 3–4× the token cost per character.
If you build multilingual products, this matters for budgeting: the same content in Japanese typically consumes 1.5–2× the tokens of its English equivalent. For high-volume system prompts, writing them in English can be a significant cost reduction — paste both versions into the counter and the difference shows up as hard numbers.
How to use it
- Open the LLM Token Counter and paste a prompt, document or code into the text area.
- Token counts appear immediately in the Claude / GPT / Gemini cards (the GPT tokenizer dictionary loads on first input).
- The per-model breakdown table below shows counts, context limits and utilization for the major models — GPT-5.5, Claude Opus, Gemini 3 Pro and more. Older GPT generations (cl100k_base) are counted separately, so generation differences are visible too.
- Character, word and line counts are tallied alongside, doubling as a general text-size check.
Everything runs in your browser; pasted text is never sent to a server, so confidential prompts and internal documents are safe to check.
The limits of estimates — and when you need exact numbers
The Claude and Gemini figures are estimates. Keep in mind:
- Don’t use them for billing math — invoice-grade numbers need each vendor’s official count
- Chat API overhead isn’t included — message structure, roles and tool definitions consume additional tokens
- Tokenizers change between generations — future models may split differently
When exact numbers are required, use Anthropic’s count_tokens API or Google’s countTokens API (both require sending your text to the API). For the everyday “roughly how big is this, without sending it anywhere?” check, a browser-local counter is the right tool.
FAQ
How accurate is the token count?
For GPT models it is exact: the official tokenizer (o200k_base) runs in your browser, so the number matches what the API charges you for. Claude and Gemini figures are estimates — neither vendor publishes a tokenizer, so the count is derived by applying a ratio to the exact GPT count based on the character mix. It is good enough for budgeting, but do not rely on it when you are right at a limit.
Does Japanese or Chinese cost more tokens than English?
Yes. English words often map to a single token, whereas CJK text frequently uses one token per one or two characters, so the same content can be two to three times larger. That is why this tool breaks the input down by character class (Latin, digits, CJK): the higher the CJK share, the closer the token count gets to the raw character count.
What happens if I exceed the context window?
The API either returns an error or, depending on the model, silently drops the oldest part. In an implementation that accumulates chat history, the typical symptom is that the assistant suddenly “forgets” the beginning of the conversation. This tool shows usage against the selected model’s context window so you can check the headroom while designing the prompt.
Is the text I paste sent anywhere?
No. The tokenizer runs in the browser, so nothing you paste into the token counter is transmitted to a server. You can measure internal documents or prompts containing API keys as-is.
Summary
- Tokens are the unit of LLM billing and context limits; knowing counts up front makes estimation and design easier
- GPT counts are exact via tiktoken (o200k_base); Claude and Gemini are estimates because their tokenizers aren’t public
- Non-English text is token-inefficient — the same content can cost 1.5–2× more than English
Wondering how big your prompt really is? Paste it into the LLM Token Counter and find out.