Production AI evaluations: sampling, review, and missed failures

A production evaluation system needs two feedback loops: one to investigate cases it flags and another to inspect cases it misses. Reviewing only alerts can improve the precision of the alert queue while leaving important blind spots invisible.

This guide proposes a sampling and review design. Its numbers are hypothetical examples, not Sentrial performance results or recommendations calibrated to your application. Choose sample sizes and thresholds based on your traffic, failure costs, and review capacity.

Define what the evaluator is deciding

Start with a specific question and unit of analysis. “Did this operation complete?” applies to an action. “Was this answer supported?” applies to a claim and its evidence. “Did this session contain any unresolved failure?” aggregates multiple events. Mixing those units makes a score difficult to interpret.

Write a rubric that includes supported success, supported failure, and insufficient evidence. Explain how reviewers should handle recovery, pending work, and contradictory events. Store the rubric version with each assessment so a later change does not silently redefine historical results.

Maintain two review queues

The first queue is targeted: flagged failures, customer complaints, new releases, expensive sessions, or high-impact actions. Its purpose is to find useful problems quickly. The second queue is a random audit of the broader eligible population, including unflagged cases. Its purpose is to learn what the targeting misses.

Do not combine the queues and call their average the production failure rate. A queue enriched for suspicious cases is deliberately unrepresentative. If you sample different traffic segments at different rates, preserve the selection probabilities and use an appropriate weighted estimate when making population claims.

Also keep a coverage report. Sessions excluded because evidence is missing or an integration is unsupported should remain visible. “We assessed 95% of eligible cases” and “we assessed 95% of all traffic” are different statements.

Read the confusion matrix carefully

Suppose, purely for illustration, a reviewed set contains 40 flagged cases. Reviewers confirm 30 failures and reject 10 flags. Alert precision within that reviewed set is 30 divided by 40, or 75%. That does not tell you recall because it says nothing about failures among the unflagged cases.

Now suppose a random audit of 100 unflagged cases finds five failures. That is evidence of misses in that audit sample. It is not automatically valid to combine those five with the 30 targeted failures and publish recall as 30 divided by 35. The two groups may have very different sampling fractions.

Report sample counts, selection method, and uncertainty alongside rates. When you lack enough observations to make a reliable estimate, say so. A precise-looking percentage from a tiny or biased sample can steer engineering work in the wrong direction.

Make disagreement useful

Have two reviewers independently label a subset of cases before seeing the evaluator’s answer. Ask them to identify the decisive evidence, not just select a label. When they disagree, distinguish ambiguous evidence from ambiguous policy.

An incomplete trace may justify “unknown.” An unclear completion contract may require a product decision. A clear contract with clear evidence and a wrong evaluator result belongs in the detector’s regression set. These problems need different fixes.

Preserve the initial labels and final adjudication. If you overwrite disagreement, you lose a valuable signal about how stable the target behavior actually is.

Introduce changes with a stable reference

Keep a fixed regression set to compare evaluator versions, and keep fresh production audits to detect drift. A fixed set alone can become unrepresentative; fresh samples alone make comparisons noisy. Use both, and avoid modifying the application, rubric, and evaluator simultaneously when you need to attribute a change.

Run a new evaluator in observation mode before allowing it to trigger operational responses. Examine false alarms, missed failures, latency, cost, and evidence coverage. Decide in advance which outcomes require human review and which can support a bounded automated action.

Close the loop with an owner

Each confirmed issue should have a route to an application fix, a new test, an instrumentation repair, or a documented decision to accept the behavior. Otherwise evaluation becomes a growing dashboard with no operational consequence.

For examples of labels that need careful boundaries, read false completion and retries and recovery. For turning reviewed incidents into reproducible experiments, use the production-to-regression workflow.