Evidence · Rote field guide

Benchmarks with the assumptions left in

Rote separates three kinds of evidence: static scorecard estimates, observed production-agent baselines, and independent research about compiled workflows. We do not present modeled savings as measured Rote benchmarks. Use rote eval --run to produce empirical results for your own tools and runtime.

Static scorecards

The included ops-report fixture estimates 452,000–1.3 million tokens before graduation and zero model tokens after because every operational node is classified as deterministic. The deal-monitor fixture estimates 922,000–2.8 million tokens before and about 1,900 after, with 75% roteness. These figures are model-based estimates with visible assumptions, not executed comparisons.

Observed inputs that calibrate estimates

The anonymized ops-report production agent averaged about 17 turns and 0.9 million cache-read tokens. The deal-monitor agent averaged about 22 turns and 1.6 million cache-read tokens. Invoice-push records two browser-agent runs of 184 and 730 turns at roughly $20 each. These observations describe source agents, not guaranteed post-graduation savings.

Independent compiled-workflow evidence

The Compiled AI paper reports 57× fewer tokens at 1,000 transactions, 450× lower median latency, 100% reproducibility for its deterministic implementation, and roughly 40× lower TCO at one million monthly transactions. Anthropic separately documents a 98.7% token reduction in one MCP code-execution example. Neither result is a universal Rote claim.

How to publish a credible comparison

Record the skill and exact input set, model names, model prices and date, tool versions, cache policy, cold and warm starts, token input and output, wall time p50 and p95, success criteria, repeated-run field agreement, and failures. Run both the original skill and graduated pipeline against the same cases.

Direct answers

Frequently asked questions

Are Rote's savings measured or estimated?

The repository includes static scorecard estimates and observed baselines from source agents. Those are labeled separately. Empirical before-and-after measurement requires rote eval --run with the real tools and runtime for a specific workflow.

Does Rote guarantee a fixed token or latency reduction?

No. Savings depend on how much of a workflow is routine, the size of tool results, retained model nodes, and runtime behavior. Rote exposes those assumptions and the remaining inference surface so teams can evaluate their own case.

Can an AI agent workflow be deterministic?

Parts of it can. Rote makes control flow explicit and moves pure functions and fixed API orchestration into testable code. Nodes intentionally kept as LLM judges or agent loops remain probabilistic. The goal is to minimize and expose the inference surface, not pretend every output is deterministic.

Graduate a workflow

Inspect the open-source CLI or run a graduation in Rote Cloud.