Last updated: July 2026
A token is the basic unit of text a large language model reads and generates — usually a subword produced by a tokenizer. As a rough rule, 1 token is about 4 English characters or 0.75 words, so 100 tokens is roughly 75 words. Both billing and the context window are measured in tokens.
If you have ever wondered why an API bill is measured in "tokens" instead of words, or why the same prompt costs more on one model than another, tokens are the answer. They are the unit every LLM actually thinks in. Once you understand what a token is and how text gets split into them, pricing, context limits, and cost control all start to make sense.
A large language model does not read raw letters or whole words. Before any text reaches the model, a component called a tokenizer chops it into tokens — the subword units the model was trained on. A token can be a whole word (the), a word fragment (token + izer), a single character, or even a chunk of whitespace or punctuation.
The model then turns each token into a number, does its math on those numbers, and generates its reply one token at a time. Everything the model reads (your prompt and context) and everything it writes (its answer) is counted in tokens. That is why token is the unit that shows up on your bill and in your context-window limit rather than "words" or "characters."
You do not need to count tokens by hand to reason about them. A few rough rules of thumb cover most English text:
These are approximations, not guarantees. Plain prose sits close to the rule; code, rare technical terms, non-English text, and heavy punctuation all push the ratio higher because they split into more tokens. When you need an exact number, a token calculator will run your text through a real tokenizer instead of the rule of thumb.
Most modern LLMs tokenize with a technique called byte-pair encoding (BPE). The idea is simple: the tokenizer learns a vocabulary of the most common character sequences in its training data, then encodes new text using the fewest, largest pieces from that vocabulary.
The practical consequences follow directly from that:
the, and, or function appear so often they earn their own entry in the vocabulary.tokenization may break into token + ization, and a truly novel string can fragment into several tokens.Different models use different tokenizers, so the exact split — and therefore the exact token count — varies from one model to another for the same input.
Take a short sentence and count it both ways. Consider:
"Tokenization turns text into tokens."
That string is 38 characters including the quotes and spaces, and 5 words. Applying the rules of thumb:
The two estimates bracket the real number, which lands around eight tokens. Note that Tokenization alone is a rare word and splits into two or three pieces, while the shorter, common words each cost a single token. The trailing period is its own token as well. This is the whole model in miniature: common pieces are cheap, rare ones are expensive, and every symbol counts.
When an API bills you, it splits tokens into two buckets, and they are almost never priced the same:
Generating text is more computationally expensive than reading it, so providers typically charge more for output — often 3 to 5 times the input rate. A verbose answer can cost several times what the prompt that triggered it did. That asymmetry is worth remembering: trimming a rambling prompt saves you input-rate tokens, but asking for a concise answer saves you output-rate tokens, which are worth more each. For a side-by-side of how the two rates differ across models, see our AI token pricing comparison.
Tokens are not an accounting curiosity — they are the two hard limits every LLM workflow runs into:
Once tokens are the unit you think in, cost control becomes concrete: fewer tokens in means a smaller bill and more room in the window. That is the core idea behind token optimization — systematically sending less text without losing meaning. For a full breakdown of how tokens accumulate in a real agent session, see our guide to AI coding agent costs.
Because everything is billed by the token, the fastest way to cut cost is simply to send less text — without losing the information the model needs:
If you want to see the numbers on your own text, our guide to reducing AI API costs walks through where the savings actually come from.
Terse compresses the prompts you send into AI coding tools and shows the per-request token cost inline — on-device, zero latency, no API calls. Cut 40-70% of the wordiness before it ever costs you a token.
See Terse for CursorA token is the basic unit of text an LLM reads and generates. It is usually a subword — a common word is often a single token, while rarer words are split into pieces. Whitespace, punctuation, and code symbols count as tokens too. As a rough rule, 1 token is about 4 English characters or 0.75 words, so 100 tokens is roughly 75 words.
Roughly 1,300 to 1,350 tokens for typical English prose, using the rule that 100 tokens is about 75 words. The exact count depends on the tokenizer and the text — code, rare words, and unusual punctuation split into more tokens than plain prose.
Generating output is more computationally expensive than reading input, so most providers charge more for output tokens — often 3 to 5 times the input rate. Your prompt and context count as input tokens; the model's reply counts as output tokens, and each is billed separately.
Send less text. Trim filler and repetition from prompts, drop irrelevant context, and reuse cached content where possible. Terse compresses the prompts you type into AI coding tools on-device, cutting token usage 40 to 70 percent with a light touch before anything is sent.