Tokenization 101

You are not billed by the word or the character — you are billed by the token, and tokens map to neither in the way you'd expect. Common words like "the" cost one token; rare or invented words get split into several. Understand the rules and you can say the same thing for meaningfully fewer tokens.

How Text Becomes Tokens

Modern LLMs use subword tokenizers (BPE-style): the tokenizer has a fixed vocabulary of common character sequences, and it greedily carves your text into the longest pieces it knows. Frequent English words are in the vocabulary whole, so "the", "function", and "return" are one token each. Rarer strings get split: "ChatGPT" is two tokens, "kubernetes" three, and a long snake_case identifier like user_authentication_status_handler can be six or more, because underscores break the match and each fragment is priced separately.

Three non-obvious rules follow from the greedy-match design:

• Whitespace counts. A leading space is usually part of the token (" cat" and "cat" are different tokens), and repeated spaces, stray blank lines, and trailing whitespace all consume extra tokens for zero meaning.
• Capitalization shifts the count. "hello" and "HELLO" tokenize differently — ALL-CAPS words often split into more pieces because the caps variants are rarer in training text.
• Punctuation and symbols add up. Every #, *, and | in formatted text is billable — the mechanics behind the markdown token tax.

"the"                          → 1 token
"ChatGPT"                      → 2 tokens
"kubernetes"                   → 3 tokens
"user_authentication_handler"  → 6 tokens
"fix   the    bug"             → extra tokens for extra spaces

The Practical Payoff: Abbreviations the Model Understands

Because common strings are cheap, the abbreviations developers already use are usually single tokens — and models read them perfectly: "function" → fn, "repository" → repo, "configuration" → config, "environment" → env, "database" → db. Applied across a technical prompt, safe abbreviations plus whitespace normalization typically cut about 20%: a verbose 2,600-token prompt drops to roughly 2,050 with identical meaning. On daily technical prompting that's worth about $70 a year, before it compounds with re-billed agent history.

The word "safe" is doing real work there. Abbreviating identifiers that appear in code, or shortening words inside quoted strings, changes meaning. A rule-based optimizer earns its keep by knowing the difference: Terse's Aggressive mode applies a curated abbreviation dictionary to prose only — code, strings, paths, and URLs are protected regions, and negations are never touched.

What This Means for Your Prompts

• Normalize whitespace. Collapse doubled spaces and stray line breaks — pure waste, zero risk. Terse's Soft mode does this (plus typo fixes) for free.
• Prefer common words. "use" beats "utilize" (1 token vs 2-3); plain phrasing is literally cheaper than fancy phrasing.
• Use standard abbreviations in throwaway prompts. fn, repo, config, env, db — instantly understood, consistently cheaper. Save the full words for text a human will read.
• Count before you assume. Token counts are unintuitive; a token calculator against your actual prompts beats any rule of thumb.

Tokenization is also why typos are surprisingly expensive: a misspelled word falls out of the vocabulary and shatters into fragments, tripling its cost while degrading the model's comprehension. That failure mode is worth its own article.

Frequently Asked Questions

Why is a rare word more expensive than a common one?

Subword tokenizers keep frequent character sequences as single vocabulary entries. Common words match whole (1 token); rare or invented words don't match and get split into several smaller pieces, each billed separately.

Do AI models understand abbreviations like fn, repo, and config?

Yes. These abbreviations are ubiquitous in the code and documentation models were trained on, so they're understood as reliably as the full words — while costing fewer tokens.

Does whitespace really cost tokens?

Yes. Repeated spaces, blank lines, and trailing whitespace each consume tokens. Collapsing them is the single safest optimization available — zero meaning change, guaranteed savings.

Further Reading

• Terse Blog — all token optimization guides
• How Typos Inflate AI Agent Costs — tokenization's failure mode
• The Markdown Token Tax — what syntax characters cost
• Token Calculator — count and price your own prompts