Prepare comparable runs
- Give the baseline and candidate distinct release labels, such as
claims-v1andclaims-v2, in the same environment. Send each label from the start of every run using the release metadata example. Keep the agent identity the same when deploying the candidate. - Open a saved signal policy and prepare its revision for evaluation. In Automatic capture and evaluation, configure that revision for your agent. Set the session and action limits, expiry and evaluation permission. To collect evaluated results, also select Check receipts automatically for a receipt policy, or enable the prepared model with its separate provider permission and budget. Saving the configuration alone does not start it.
- Start the configuration before collecting the baseline. It selects session records created after activation, once they complete and settle; earlier runs are not backfilled. Check its status and capture history on the policy page. Capture and the chosen evaluator must be enabled in your deployment.
- Collect the baseline, deploy your change, then collect the candidate. Use the same capture configuration, policy and evaluator for both. Allow the runs and their evaluations to finish before saving the comparison.
Save a comparison
In Release reports, choose your agent and its Saved signal measurement. Enter the environment and the exact, case-sensitive baseline and candidate labels. Enter two ended, non-overlapping windows in UTC, with the baseline first. The window includes runs starting at From and excludes runs starting at Until. For example, 10:00–11:00 includes a run starting at 10:00, but not one starting at 11:00. These fields use UTC even when your computer uses another time zone. Open Comparison requirements to review the defaults: at least 90% resolved coverage and ten resolved runs per release. These thresholds determine whether a difference is shown; they do not establish statistical significance. The two windows together may contain at most 100 runs for the agent, including runs with another or missing release label. Narrow the windows if the selection is too large. Click Save comparison. The report is available under Saved reports for the selected agent; the list shows its latest twenty comparisons.Read the result
Each release shows detected failures divided by resolved runs, plus the counts needed to understand what was left out:
The difference is candidate minus baseline, in percentage points. For a
hypothetical baseline with four detected failures among eighteen resolved runs,
and a candidate with one among eighteen, the rates are 22.2% and 5.6%: a difference
of −16.7 percentage points. The comparison still needs to meet its coverage and
run-count requirements. Traffic composition and detector errors can affect these
observations, so the difference alone does not establish that the release caused
an improvement.
Use Runs included in the report to inspect excluded runs and open their original
sessions. Review the tool results and assistant responses behind a detected issue.
Measurement and saved identity shows which signal, policy and evaluator versions
produced the comparison.
Resolve an incomplete comparison
Later arrivals and evaluations do not update an existing report. Save a new
comparison when you want a new snapshot of the evidence.