What belongs in an AI production-readiness review?

production · evaluation · security · 2026-09-15

A review checklist for evaluation, access, observability, cost, and operations — the parts demos skip.

A production-readiness review for an AI workload is not a model bake-off. It is a review of the system that will run when the people who built the demo are not in the room.

Use this as a conversation with engineering, security, and the workflow owner — not as a score that replaces judgment.

Evaluation

Held-out cases exist and are re-run on every prompt, model, or tool change.
Success is defined without model self-report.
Failure modes are named (abstain, escalate, retry, block).

If you cannot regress the system, you cannot change it.

Observability

Prompts, tool calls, and outputs are logged with retention and redaction.
You can trace an incident to input, model version, and tool path.
There is a way to sample production traffic into the eval set.

Security and access

Data classification matches the provider and the region.
Secrets never enter the context window.
Tools are least-privilege; write actions are scoped and audited.

“We’ll lock it down after the pilot” is how pilots become production without a review.

Human controls

Irreversible actions require approval.
The human sees enough context to decide, not just “the agent recommends yes.”
There is a path to disable the agent and continue the workflow manually.

Deployment and reliability

Versioned artifacts: prompt or graph, tools, model IDs, retrieval config.
Rollback exists.
Timeouts, retries, and fallbacks are explicit.
The system degrades into a known state, not a stuck loop.

Data

Source-of-truth systems are named.
Retrieval corpora have owners, freshness, and access control.
You know what happens when the document is wrong or stale.

Cost and operations

Budget and a spend kill switch.
A named owner and a runbook.
An isolation switch that does not take adjacent systems down.

How to use the review

Walk the list against one workload. Anything unanswered is a gap, not a “later.” If most items are gaps, you still have a prototype — which is fine, if you call it that.

We publish an ungated production-readiness assessment that scores these dimensions. Use it as a diagnosis, then decide whether a Production Sprint is the right way to close the gaps in your environment.