Your Prompt Didn't Change. Your Token Bill Did: The Tokenizer Tax

You upgrade to a newer model. The prompt is the same text, the per-token price looks the same or lower, and the invoice goes up anyway. The usual suspect is the model. The less-noticed one is the tokenizer: the piece that turns your text into the tokens you are billed for.

This article follows that cost through the documentation: what Anthropic says its newer tokenizer does, what it does to the price of the same text, why counts differ between providers, and how to measure your own.

How to read it: every claim about tokenizers and prices comes from the providers' documentation, the pricing page, and one 2026 paper, which I read on October 14, 2026. The cost examples are my arithmetic. I ran no live token counts, so I have no measurement of my own. Where a page was silent, I say so.

A token is not a unit of text

A tokenizer splits text into pieces from a fixed vocabulary, and each model family has its own. Billing, rate limits and context windows are all counted in those pieces. So a token is a unit defined by one model's vocabulary, not a fixed amount of text.

A 2026 paper, "How Long Is a Piece of String?", compares ten tokenizers (Claude, DeepSeek, Gemini, Grok, GPT, Mistral, Llama 4, Reka, Jamba and Qwen) on eight kinds of text. Its findings:

• Compression for the same kind of text varies across tokenizers by roughly 20 to 50 percent, and by about 100 percent for emoji and digit strings.
• Within one tokenizer, the gap between the worst case (emoji) and the best (code and essays) is nearly tenfold.
• The common rule of thumb of 0.75 words per token is too simple. For Claude and Gemini the average is above 0.75 for the most frequent words. For DeepSeek and Mistral it is closer to 0.6, and for rare words it falls to 0.35 to 0.45.
• One long word, "antidisestablishmentarianism", takes 9 tokens on Claude and 5 on Gemini or Grok.
• Reported context limits are not comparable: equal limits in tokens hold different amounts of text.

The authors conclude that tokenization varies significantly across models and domains, which makes naive interpretation of token counts problematic.

The providers say the same in softer language. OpenAI's help page says the same text can produce different token counts depending on the model, its encoding and the language, and points to its tokenizer tool and tiktoken for exact counts. Gemini's page gives a rule of thumb of about 4 characters per token and offers a count_tokens call that counts the input only.

What Anthropic changed

Anthropic's token counting page states it directly: Claude 4.7 and later models use a newer tokenizer, and the same input text produces approximately 30 percent more tokens than on earlier models. The exact increase depends on the content and workload shape. The Opus 5.5 migration guide gives the range: roughly 1x to 1.35x as many tokens, up to about 35 percent more, varying by content.

Three details matter:

1. Billing follows the new counts. The page says usage and billing on these models reflect the new tokenizer's counts, and says not to reuse token counts measured on an older model to estimate cost or context fit.
2. The pricing page lists the split. Claude Sonnet 4.6 and earlier use the previous tokenizer. Claude 4.7 and later use the new one.
3. It reaches beyond the invoice. The migration guide tells you to give max_tokens additional headroom, including compaction triggers, and to re-test any code that estimates tokens client-side or assumes a fixed token-to-character ratio.

Anthropic does not publish the multiplier per content type on these pages. Some third-party posts report that code and structured data inflate more than prose. I did not use those figures, because I could not trace them to a primary source.

What it does to the price of the same text

The pricing page lists each model's price per million input tokens. Take the same text that costs 1 million tokens on a pre-4.7 model, and apply the documented roughly 30 percent. All figures below are my arithmetic on input tokens only.

Move	List price per million input tokens	Same text, effective	Change
Opus 4.6 to Opus 4.7	$5 to $5	$5 to $6.50	+30%
Opus 4.6 to Opus 5.5	$5 to $4	$5 to $5.20	+4%
Sonnet 4.6 to Sonnet 5.5	$3 to $2	$3 to $2.60	-13%
Haiku 4.5 to Haiku 5.5	$1 to $0.10	$1 to $0.13	-87%

Haiku 5.5 is shown at its under-100,000-token prompt price. Above that it is $0.50.

Read it as a monthly bill. A workload of 100 million input tokens a month on Opus 4.6 costs $500. The same text on Opus 4.7 is about 130 million tokens at the same $5, so $650. On Opus 5.5 it is 130 million at $4, so $520. On Sonnet, $300 on 4.6 becomes about $260 on Sonnet 5.5.

Two lessons:

• Opus 4.6 to 4.7 is a pure tax. The list price did not change and the same text costs about 30 percent more. This is the case the headline describes.
• Where the price was cut, the tokenizer takes back part of the cut. The cut was 33 percent on Sonnet input, but the effective saving is about 13 percent.

Because the multiplier is a range of 1.0 to 1.35, the answer depends on your content. Opus 5.5 costs more than Opus 4.6 for the same text once the multiplier passes 1.25 (5 divided by 4). At the top of the documented range, 1.35, that is $5.40 against $5, or 8 percent more. Sonnet 5.5 stays cheaper than Sonnet 4.6 across the whole documented range, because the break-even is 1.5.

Cached reads scale the same way. A cache read on Opus 4.6 and 4.7 is $0.50 per million tokens, but the same cached text is about 30 percent more tokens on 4.7, so about $0.65 for the same text. Caching lowers the price per token. It does not lower the token count.

I did not find a documented multiplier for output tokens. The pricing page says the new tokenizer produces about 30 percent more tokens for the same text, which would apply to a reply too, but I did not see an output-specific figure, so I left output out of the table.

What else counts in tokens

Anything measured in tokens changes with the tokenizer:

• Context window. A window of fixed size in tokens holds less text. At 1.3x, it holds about 77 percent as much (1 divided by 1.3).
• Rate limits. Anthropic's input and output limits are per minute in tokens, so the same traffic may use more of them. That is my inference from the limits being token-based. I did not test it.
• Compaction and truncation thresholds. If you trigger them at a token count you chose on the old model, they now fire at about 23 percent less text.
• Client-side estimates. Any code that estimates tokens as characters divided by four is now wrong in a way that depends on content.

How to measure your own

Anthropic's token counting endpoint is free. Its limits are separate from message creation, and it counts under the tokenizer of the model you name. The docs suggest counting the same request twice, once with your current model and once with the one you plan to move to.

import anthropic

client = anthropic.Anthropic()
request = dict(
    system=YOUR_SYSTEM_PROMPT,
    tools=YOUR_TOOLS,
    messages=YOUR_TYPICAL_MESSAGES,
)

for model in (CURRENT_MODEL, TARGET_MODEL):
    n = client.messages.count_tokens(model=model, **request).input_tokens
    print(model, n)

Three caveats from the same page:

• The count is an estimate and may differ by a small amount from actual usage.
• Counts may include tokens that Anthropic adds for system optimizations, and you are not billed for those.
• It does not use caching. Cache read and write pricing applies only when you create a real message.

The endpoint also counts tools, images and PDFs, so you can measure your real request, not just a text sample. It returns an error for server tools such as web search and for the MCP connector. For those, read the usage object on a real response.

What to do about it

1. Recount before you migrate. Run your real prompts through count_tokens on both models and compare. Do not reuse old counts.
2. Compare cost per task, not per token. Per-token price alone misleads in both directions, as the table shows.
3. Add headroom. Raise max_tokens and move compaction triggers, as the migration guide says.
4. Check your content type. If most of your traffic is short prose, you may be near 1.0x. If it is code or structured data, assume the top of the range until you measure.
5. Do the same on other providers. OpenAI points to tiktoken and Gemini to count_tokens. If you route between providers, one shared token budget will not hold the same amount of text on each.
6. Re-test token-based assumptions. Search your code for division by four and for hard-coded token thresholds.

What I could not verify

I have no API key in this environment, so I ran no live token counts and cannot confirm the multiplier on your workload or mine. The 30 percent and the 1x to 1.35x range are Anthropic's figures, and the table is my arithmetic on them. I did not find a documented output-token multiplier, a per-content-type breakdown from Anthropic, or whether rate limits are applied after the new counts. For other providers I read only the general token pages, not model-specific release notes, so I make no claim that OpenAI or Google changed a tokenizer recently. The paper's per-domain values are in a figure I could not read as text. Prices, models and tokenizers change, so recheck before relying on any number.

Sources, all accessed October 14, 2026: Anthropic: Token counting; Pricing; Claude Opus 5.5 migration guide (platform.claude.com/docs). OpenAI: What are tokens and how to count them (help.openai.com). Google: Understand and count tokens (ai.google.dev). Roberts, Han and Albanie, "How Long Is a Piece of String? A Brief Empirical Analysis of Tokenizers", arXiv:2601.11518.

When did you last recount a prompt after switching models, and how far was the new number from the old one?

#AI #LLM #AIEngineering #Tokens

---

Work with me: I take 1:1 calls on mobile architecture reviews, mobile and AI questions, and startup tech consults. Book a slot on Topmate: topmate.io/sairamkotha