Last updated: July 2026
Prompt compression is reducing the token count of a prompt or its surrounding context while preserving meaning and the model's output quality. It trims filler, redundancy, and low-information tokens so the same instruction costs fewer tokens — lowering price, freeing context-window space, and speeding up responses.
Every token you send to a language model is billed, occupies the context window, and adds to the time the model spends reading before it answers. Prompt compression is the practice of shrinking that input without changing what you actually mean. Below is what it is, the two families of techniques, the trade-offs to watch, and why it matters more the longer your session runs.
Prompt compression is any technique that reduces the number of tokens in a prompt, or in the context supplied alongside it, while keeping the meaning intact enough that the model produces the same — or effectively the same — output. The goal is not to write a shorter message for its own sake; it is to remove the tokens that carry no information the model needs, so the ones that remain do all the work.
That distinction matters. A good compression pass is measured by two numbers at once: how many tokens it removed, and how little the output changed. Removing half the tokens is worthless if the answer degrades. The whole discipline lives in the tension between those two figures. It sits alongside the broader practice of token optimization, which covers cost and context management more generally — compression is the specific act of making a given piece of text smaller.
The first family works on the surface text with deterministic rules. It never asks a model what to cut; it applies fixed transformations that are safe by construction. Typical operations include:
Because every rule is fixed and reversible in principle, lexical compression is predictable — the same input always yields the same output, and it never rewrites your meaning. This is exactly what Terse does, entirely on-device, with no model call in the loop. There is a long lineage here: telegraph-style compression — writing in the terse register people once used because they were charged per word — is the human ancestor of the same idea. You strip the connective tissue and keep the payload.
The second family uses a small language model to decide what to drop. Instead of fixed rules, it scores each token by how much information it carries and removes the low-information ones. The best-known example is LLMLingua (Microsoft, EMNLP 2023), which uses a compact model to estimate token-level perplexity and prune the least essential tokens before the prompt reaches the large model. On some tasks it reaches up to roughly 20x compression while keeping output quality close to the uncompressed baseline.
The strength of model-based compression is that it adapts to content — it can compress a verbose explanation hard while leaving a dense technical passage nearly untouched. The cost is that it requires running a model, which adds latency and its own compute, and because it is probabilistic rather than deterministic, the result can vary and occasionally cut something that mattered. For long, information-dense contexts where the savings are large, that trade is often worth it.
Two related approaches attack the context rather than the prompt itself. Selective context prunes irrelevant material before it is ever sent — filtering retrieved documents, dropping files the task does not touch, and keeping only the passages that bear on the current question. It is compression by omission: the cheapest token is the one you never send.
History compaction targets the growing conversation. In a long agent session, the transcript is re-sent every turn, so summarizing older turns into a compact recap — replacing a 4,000-token tool dump from ten turns ago with a two-sentence summary — keeps the running context from ballooning. Both techniques recognize that in agent workflows the context, not the single prompt, is where most tokens actually live.
All compression sits on one dial: how aggressively you cut versus how much nuance you risk losing. Push too hard and you strip the qualifiers, edge cases, and domain terms that changed the answer. The failure looks subtle — the model still responds confidently, just to a slightly different question than you asked.
Good tools defend against this by protecting the tokens that must survive. Code blocks, identifiers, file paths, numbers, error messages, and key domain terms are held verbatim; only the connective padding around them is trimmed. Deterministic lexical methods are the safest here because they never paraphrase — they can only delete text they are certain is redundant. The rule of thumb: compress the wrapping, never the payload.
Three concrete payoffs justify the effort:
The effect compounds hardest in agent sessions. Because the context is re-sent on every turn, a prompt trimmed once is paid down on every turn it rides along on — so even a modest per-prompt saving multiplies across a long run. That is the mechanism behind runaway AI coding agent costs, and it is exactly where a small, deterministic compression pass earns its keep.
Terse applies deterministic lexical compression to the prompts you send into AI coding tools — trimming filler, politeness, and redundancy while preserving code and key terms. No API calls, no latency, and per-request token savings you can see.
See Terse in ActionPrompt compression is reducing the token count of a prompt or context while preserving its meaning and the model's output quality. It removes low-information words, filler, and redundancy so the same instruction fits in fewer tokens — lowering cost, freeing context-window room, and speeding up responses.
There are two main families. Lexical or rule-based compression deterministically removes filler, politeness, and hedging, shortens phrases, strips redundant markdown, and abbreviates — this is what Terse does on-device. Model-based compression, such as LLMLingua (EMNLP 2023), uses a small model to drop low-information tokens, reaching up to roughly 20x compression on some tasks.
It can if applied too aggressively, because nuance and key terms may be dropped. Good compression preserves code, identifiers, numbers, and domain terms while removing only redundant text. Deterministic lexical methods are safest because they never rewrite meaning, only trim padding.
Fewer tokens mean lower cost per request, more room in the context window for the actual task, and faster responses because the model processes less input. In long agent sessions, where context is re-sent every turn, even small per-prompt savings compound significantly.