How to estimate inference cost before production

Inference price lists are not a cost model. A cost model is how often the workflow runs, what happens on each run, and who still has to look at the output.

Start from volume, not from $/1M tokens

Write down:

• Cases per day or per week
• Peak, not only average (retries cluster)
• How many model calls per case (planner, critic, embeddings, fallback)
• Input and output tokens per call, measured on real documents, not marketing context windows
• Tool calls that themselves cost money or time

Then:

monthly inference ≈ cases × calls_per_case × (input_tokens × pin + output_tokens × pout)

If you do not have measurements, instrument a prototype for a week. Guessing token counts from the UI is how teams miss an order of magnitude.

The cheap model is not the cheap system

A smaller model that fails 20% of the time and triggers a retry, a larger fallback, or a human rewrite can cost more than a larger model that finishes in one pass.

Price completed successful work, including:

• Retries and agent loops
• Retrieval / embedding refresh
• Guardrail or judge models
• Human review minutes × loaded cost of that reviewer

Human time usually dominates until the workflow is both frequent and highly automated.

What to include besides tokens

• Provider minimums, provisioned throughput, or reserved capacity
• Logging and trace storage (verbose prompt logs are not free)
• Evaluation runs in CI (they should exist; they also spend)
• Failed runs that still consume tokens

Break-even, stated honestly

Compare against the current cost of the workflow: people, delay, error rate. If you cannot name the current cost, you cannot claim savings.

A useful pre-production question: at expected volume, does inference plus review cost less than the status quo by enough to pay for engineering and operations? If the answer depends on an unsupported productivity percentage, stop.

What to do with the number

Treat it as a design constraint. It tells you whether to:

• Shrink context and tool surface
• Replace an agent loop with a deterministic path
• Add caching or batching
• Keep a human in the loop (and budget for it)
• Not build this at all

We will add an economics calculator to /ai/tools. Until then, a spreadsheet with measured tokens beats a vendor calculator with hopeful defaults.