Production evaluation is a decision system. It should tell a team whether to release, what to fix, which users need protection, and when a model or prompt change created a regression.
A handful of attractive examples cannot do that. The evaluation population must represent the tasks, users, source conditions, and failure costs of the workflow.
Define the unit of success
Begin with the user’s job and the downstream decision. A support draft may need correct policy, complete required steps, appropriate tone, and accurate citations. An extraction system needs field-level correctness, abstention, and schema validity. An agent also needs correct tool selection and safe action behavior.
Separate dimensions instead of collapsing them too early into one score. A system with excellent style and poor evidence grounding should fail clearly.
Build a representative evaluation population
Sample normal work, difficult edge cases, known historical failures, adversarial inputs, missing information, and cases that should be refused or escalated. Preserve meaningful metadata so results can be segmented.
- Task and user segment
- Source or document condition
- Language and input length
- Risk or consequence class
- Expected tool or evidence path
- Known ambiguity and acceptable alternatives
Combine deterministic, model, and human evaluation
Use deterministic checks for schema, citations, permissions, tool arguments, and prohibited states. Model-based judges can scale nuanced comparisons but need calibration, position-bias controls, and human audits. Domain experts remain necessary where correctness depends on professional judgment.
Record disagreements rather than forcing false precision. They often expose an unclear rubric or a workflow where multiple outcomes are acceptable.
Turn results into release gates
Define minimum performance for critical slices, maximum severe-failure counts, latency and cost limits, and rollback triggers. Store inputs, outputs, model and prompt versions, tool traces, and grader versions so a result can be reproduced.
Build the evaluation set from operating reality
Start with representative traffic, known failures, policy-sensitive cases, and deliberately difficult counterexamples. Label the population dimensions that can change the decision: customer segment, document type, language, tool, risk level, and ambiguity. Averages without slices hide regressions that affect a small but consequential group.
Separate reference answers from scoring rules. Some tasks allow exact or structured comparison; others require criteria such as completeness, evidence use, tone, or policy compliance. Human judgment should follow a written rubric with examples of acceptable disagreement. Otherwise evaluator preference becomes an unstable hidden requirement.
Connect evaluation to release governance
Define which changes trigger which suites: model upgrade, prompt edit, new retrieval source, tool schema, policy, or user-interface change. Establish blocking thresholds for critical failures and review thresholds for tradeoffs. A release gate should show changed results, confidence, affected slices, and an explicit decision owner.
Online monitoring complements rather than replaces offline evaluation. Production signals find distribution change and new failure classes; curated suites make those failures reproducible. Create a routine for turning incidents, escalations, and reviewer corrections into durable tests. The evaluation asset should become more representative as the product operates.
Executive decision record
The decision is what measured behavior is sufficient to release a specific system change to a defined population and authority level. Write that decision before selecting a model, vendor, framework, or implementation pattern. A written boundary keeps technical exploration connected to the operating outcome and makes it possible to explain why the organization advanced, revised, or stopped the work.
Approval should depend on representative and adversarial cases, written scoring rubrics, critical slices, baseline comparisons, uncertainty, and an explicit release record. The evidence does not need to remove every uncertainty, but it should address the uncertainty capable of changing value, architecture, risk, or ownership. Record the baseline, assumptions, unresolved questions, and the person accepting the next stage.
Failure boundary and operating ownership
The central failure to guard against is optimizing one aggregate score while rare policy, safety, or customer-impact failures remain invisible. Treat that condition as a testable scenario. Define how the system detects it, what users experience, which action is prevented or reversed, and what evidence reaches the person responsible for recovery.
Long-term accountability sits with the product owner who accepts behavior, supported by domain reviewers and engineers who maintain the suite and evidence pipeline. Supporting specialists can provide platforms, research, review, or delivery capacity, but they cannot substitute for an owner who controls policy and operating change. Name that owner before production and include the ownership path in release evidence and incident procedure.
A practical 90-day application plan
During the first 30 days, convert representative and adversarial cases, written scoring rubrics, critical slices, baseline comparisons, uncertainty, and an explicit release record into a bounded evidence plan. Assign each artifact to a named contributor, identify the representative inputs required, and agree on the comparison baseline before implementation expands. The objective of this period is to expose the assumption most likely to invalidate the work while the cost of changing direction is still low.
During days 31 through 60, build or instrument the smallest complete workflow that can support the decision about what measured behavior is sufficient to release a specific system change to a defined population and authority level. Include the real data and authorization path where feasible, record exceptions, and review difficult cases with the people who own the underlying process. Resist adding breadth until the team can explain the measured behavior of this narrow slice.
During days 61 through 90, test the boundary represented by optimizing one aggregate score while rare policy, safety, or customer-impact failures remain invisible. Exercise degraded dependencies, ambiguous inputs, recovery, and handoff rather than demonstrating only successful cases. End the period with a written advance, revise, or stop decision that cites evidence, residual exposure, expected operating cost, and the next authority boundary.
The review should be accepted by the product owner who accepts behavior, supported by domain reviewers and engineers who maintain the suite and evidence pipeline. That group should confirm not only that the system can work, but that ownership, support capacity, monitoring, and change control are credible. If those conditions are absent, the responsible outcome is another bounded learning stage rather than an unsupported production commitment.
Practical checklist
- Write acceptance criteria before tuning the system.
- Include cases that must refuse or escalate.
- Segment by risk and operating condition.
- Calibrate automated graders against expert review.
- Track severe failures separately from averages.
- Version every dependency that changes behavior.
- Run regressions before model, prompt, data, or tool releases.
Engagement scenario
An insurance-document assistant performs well overall but omits a required disclosure on one policy type. Slice-level release gates expose the issue even though the aggregate score improved. The release is held until the disclosure path passes deterministic and expert review.
Continue reading
- [AI pilot to production](/ai-pilot-to-production-roadmap)
- [Observability for AI agents](/ai-agent-observability-production)
- [AI risk assessment with NIST](/ai-risk-assessment-nist-rmf)
Sources and further reading
- [NIST AI Resource Center](https://airc.nist.gov/)
- [NIST AI RMF](https://www.nist.gov/itl/ai-risk-management-framework)