AI Production Readiness Assessment

0 of 14 answered

1. 1 · evaluation

   Do you have an evaluation set of real tasks, held out from prompt-tuning?

   Chat demos are not a regression suite. Production needs cases you can re-run after every change.

   NoPartialYes

2. 2 · evaluation

   Can you score success without asking the model whether it did a good job?

   Self-grading inflates confidence. Prefer checkable outputs, golden answers, or a human rubric.

   NoPartialYes

3. 3 · observability

   Do you log prompts, tool calls, and outputs with a retention and redaction policy?

   You cannot debug or audit what you did not keep — and you should not keep secrets forever.

   NoPartialYes

4. 4 · observability

   Can you trace a production failure to a specific input, model version, and tool path?

   “The agent was weird today” is not an incident report.

   NoPartialYes

5. 5 · security

   Is input data classified, and is the model or provider allowed to see that class of data?

   A working prototype that ships regulated or customer data to the wrong endpoint is not a win.

   NoPartialYes

6. 6 · security

   Are secrets and credentials kept out of prompts, context windows, and tool results?

   Models and logs leak. Treat prompt contents as an exfil path.

   NoPartialYes

7. 7 · access

   Does the agent have least-privilege access to tools, APIs, and data — per action, not a god token?

   Tool-calling agents fail open if every capability sits behind one credential.

   NoPartialYes

8. 8 · human-controls

   Is there a human approval step for irreversible or high-impact actions?

   Drafts can be cheap. Sending, paying, deleting, or changing access should not be.

   NoPartialYes

9. 9 · deployment

   Is the workload a versioned service with rollback, not a notebook or a chat thread?

   If you cannot pin prompts, tools, and model IDs, you cannot ship a change safely.

   NoPartialYes

10. 10 · data

    Do you know lineage, freshness, and access controls for retrieval or source data?

    RAG quality is a data problem first. Stale or over-permissioned corpora become silent failures.

    NoPartialYes

11. 11 · reliability

    Are timeouts, retries, and fallbacks defined for model and tool failures?

    Providers fail. Tools fail. The workflow still has to degrade into something an operator understands.

    NoPartialYes

12. 12 · cost

    Do you have a budget, token or call accounting, and a way to stop spend?

    Retries, agents, and long context turn a cheap demo into an unbounded bill.

    NoPartialYes

13. 13 · operations

    Is there a named owner and a runbook for this workload?

    If nobody is on the hook when it misbehaves at 2 a.m., it is not in production.

    NoPartialYes

14. 14 · operations

    Can you disable or constrain this system without taking adjacent systems down?

    A feature flag and a kill switch matter more than another dashboard.

    NoPartialYes