Where do the tokens actually go?
In a multi-turn LLM application, most tokens are not the answer. They are the system prompt, the tool schemas, the conversation so far, and the full body of every prior tool result, re-sent on each turn because that is how a conversation works. Output tokens are usually the smaller share. Any optimization plan that starts from the visible response rather than the resent context is optimizing the wrong number, so measure the input side first.
How far does prompt and context hygiene get you?
Trim the system prompt to what the current task needs. Prune reference documents and few-shot examples that no longer earn their place. Filter tool results before they enter context instead of passing whole payloads through the model. Summarize or truncate history once it stops informing decisions. These are real savings and the right first step, but they are bounded, and they pull against quality: the references and examples you cut are often the reason the application got good.
What does prompt caching change?
Prompt caching lets the provider reuse a static prefix, so the instructions, schemas, and references you resend every turn are charged at a reduced rate rather than full price. It is worth enabling almost everywhere. It is also strictly a discount: cached input still costs money, cache entries expire, and caching does nothing about the number of model turns a multi-step procedure takes. A workflow that runs twenty turns with caching is still a twenty-turn workflow.
When does model routing pay off?
Not every step needs the largest model. Classification, extraction, and reformatting often succeed on a smaller, cheaper model, with escalation to a stronger one when confidence is low or the input is unusual. Routing lowers the unit price of the steps that remain model calls. Its limits are the same as the other pricing levers: every routed step still pays for inference, and the router itself adds complexity and an evaluation burden that grows with each model in the cascade.
When should a step stop being a model call at all?
A mature workflow is mostly settled steps: validating a field, routing on a threshold, formatting a payload, calling APIs in a fixed order. Code performs these exactly, at zero tokens, every run. Moving deterministic work out of inference is the only lever whose savings grow with usage, because a removed call is removed from every future run. Rote applies this systematically: it compiles a proven agent skill into typed code and keeps model calls only for the steps that genuinely require judgment.
Independent Compiled AI research (arXiv 2604.05150): 57× fewer tokens at 1,000 transactions
Both describe specific studies, not universal Rote guarantees.
How do you measure token efficiency honestly?
Count runs per month and average input and output tokens per run from real logs, not estimates. Record which model, which prices, and the date, because prices move. Split each run's tokens into decisions that genuinely varied and procedure that did not, since only the second share is removable. Then apply the levers in order of effort: hygiene and caching first, routing where quality holds, and compilation for the workflows where repetition has become the dominant cost.
Frequently asked questions
How do I reduce token consumption in an LLM application?
Apply four levers in order: shrink the context each call carries, enable prompt caching so resent context is charged at a reduced rate, route simple steps to cheaper models, and move deterministic steps out of inference into code. The first three lower the price of repetition; the fourth removes it, which is why it matters most for workflows that run often.
Is prompt caching enough to control LLM costs?
It helps and you should enable it, but it discounts re-reading rather than removing it. Cached input still costs money, entries expire, and the turn count of a multi-step workflow is unchanged. For a procedure that has stopped changing, removing the model calls for its settled steps is the larger and more durable lever.
When should I compile a skill?
Keep one-off exploration in an agent. Compile a skill after the procedure is proven, repeats often, and needs lower cost, faster execution, regression tests, explicit approvals, or reliable retries.
Compile a workflow
Inspect the open-source CLI or run a compilation in Rote Cloud.