GridArray Editorial
Production readiness checklist
A practical checklist for engineering teams deciding whether a working AI agent or agent-facing product is ready to operate reliably.
Direct answer
Production readiness is a system judgment, not a model score.
A working agent proves that a workflow can produce a useful result. Production readiness asks whether the data, access, tools, execution, evaluation, and operations behind that result can be trusted when real users, permissions, and failures show up. Use this checklist on one workflow. Mark each item ready, at risk, or blocked. The blocked items on the critical path are the engineering work. This page does not replace a readiness scorecard, and it does not provide an architecture design.
Framework
Six checks before production traffic
Run the six topics below with the people who own the workflow, the data, and the release decision. Keep the scope on the systems that this workflow actually calls. A green checklist for a demo path is not evidence that the wider estate is ready.
Data quality and provenance
Agents repeat whatever the sources assert. If two systems disagree about a customer, a document, or a status, the agent will treat the conflict as knowledge unless the workflow has an authority rule and a source trail.
- Name the workflow and the records or documents it must trust.
- Identify the system of record for each fact the agent uses.
- Write the rule for conflicting or duplicate records.
- Attach a source and a timestamp to retrieved context.
- Reject stale channels, drafts, and unowned files as if they were authoritative.
- Decide who reviews a result when provenance is missing.
Agent access and permissions
A broad connector feels fast in a pilot and unsafe in production. Access should name the caller, the identity in force, and the smallest set of reads and actions that workflow needs.
- Bind each call to a named identity, not a shared superuser account.
- List the systems the workflow may read and the actions it may take.
- Scope permissions to that workflow rather than to a department-wide role.
- Block access to records the caller is not allowed to see.
- Record who approved an exception when access is wider than the default.
- Remove tools that exist only because a backend exposes them.
API and tool reliability
Tool calls fail in ordinary ways: timeouts, partial payloads, changed schemas, and rate limits. A production surface makes those failures explicit instead of letting the agent improvise a next step.
- Define the contract for each API or tool the workflow calls.
- Validate inputs before a write or another irreversible action.
- Specify timeout, retry, and idempotency behaviour for each call.
- Detect schema changes before they reach production traffic.
- Return structured errors the workflow can branch on.
- Keep a human path for actions the contract cannot safely complete.
Execution and workflow recovery
Multi-step agents accumulate state. If a run dies on step four, the system needs to know what already happened, what can be retried, and what must stop.
- Persist the step, inputs, and outputs of an in-flight run.
- Separate retryable failures from failures that must halt the workflow.
- Prevent a retry from repeating a payment, message, or other side effect.
- Expose a resume or cancel action for operators.
- Keep review checkpoints where a person must approve the next step.
- Test a crash in the middle of the workflow, not only the happy path.
Agent evaluation
A handful of impressive transcripts is not an evaluation set. Production evaluation checks the tasks the business cares about, including bad context, refused actions, and partial tool failure.
- Write the tasks that define success for this workflow.
- Include cases with conflicting sources, missing permissions, and tool errors.
- Score task success, context quality, and unsafe actions separately.
- Re-run the set when prompts, retrieval, or tools change.
- Block release when a critical task regresses.
- Keep the cases and results where the owning team can inspect them.
Observability and operational readiness
After release, someone has to see a bad run, understand why it happened, and stop it from spreading. That work is part of the production system, not a dashboard added later.
- Trace each run across model calls, retrieval, and tools.
- Alert on failure rate, latency, cost, and refused or unsafe actions.
- Give operators a way to inspect inputs, sources, and tool results.
- Define who responds when a production run fails.
- Record a rollback or feature-flag path for a bad release.
- Review incidents and feed the findings back into evaluation.
Diagnosis
What teams notice, and what is actually missing
Symptoms
- The demo works for the founder and fails on a customer record.
- The same question returns a different answer from a different system.
- A tool timeout turns into an invented next step.
- Nobody can tell which run failed, or which source it used.
Root causes
- No authority rule or provenance on the context the agent uses.
- Permissions copied from a broad role or a shared service account.
- Tool contracts without timeouts, validation, or idempotent retries.
- No saved run state, evaluation set, or operator path after release.
Implementation
How to use the checklist
Hold one working session with the engineering owner and the person who can approve a release. Walk a single workflow from the first source it reads to the last action it takes. Mark every item ready, at risk, or blocked. Sequence engineering around the blocked items that sit on the path to production. Bring that marked checklist to an AI Production Readiness Call if you want a one-page scorecard and a paid plan for the gaps.
- Limit the review to one live workflow or serious pilot.
- Write the blocker in the words the team already uses.
- Separate data gaps from access, execution, and operations gaps.
- Do not expand into estate-wide cleanup during the session.
- Treat the result as a priority list, not a design document.
Boundaries
Where this checklist stops
- It does not score an entire company data estate.
- It does not design architecture or implementation for free.
- It does not replace security review, cost control, or a readiness scorecard.
- Greenfield ideation and standalone connectors without a real workflow stay out of scope.