Continual learning for agents: a production feedback loop

A production failure should help you improve the next release. That starts with a specific question: what did the agent say happened, and what actually happened?

Sentrial’s current learning loop is built around reviewed evidence and versioned changes. Capture a run, inspect a failure, keep it as a regression case, compare a repair and watch the next release. The change might be in an adapter, a prompt, a tool contract or a model choice. Updating model weights is a separate engineering step with its own data and evaluation requirements.

Start with evidence you can inspect

In our recorded insurance example, an agent reported that a claim had been submitted. The sandbox submission service had rejected it. An adapter replaced the rejection with success: true, which is what the model saw.

That gives an engineer a concrete defect to investigate. Preserve the service’s original result, the observation passed to the model, the agent’s response and the identity of the attempted action. A successful tool call alone cannot establish that the requested business action completed.

These recordings use real model responses and synthetic claim data. They show a development workflow, with no real insurance submission or customer result. The false-completion guide includes the receipt and the repaired response.

Keep the failure, then add controls

One example gives you a starting point. A useful test set also checks the cases a repair could break:

• Explicit rejection: Does the agent report the rejection accurately?
• Ordinary successful submission: Can it still recognize a valid receipt?
• Timeout after the service committed: Does a status lookup recover the actual outcome?
• Work still pending: Does the response make the unfinished state clear?
• Missing or conflicting evidence: Does it preserve uncertainty?
• Success for another action: Does it avoid using the wrong receipt?

Define these expectations before comparing versions. Keep the original failure as a regression case and use separate, previously unseen examples to assess generalization. Repeatedly editing a prompt against the same known cases measures development progress, not performance on new traffic.

Compare the change you actually made

In the recorded repair, we changed both the adapter and the instructions. The adapter preserves the original receipt; the instructions require checking that evidence before reporting completion. The revised response accurately reports the rejection.

The claim itself remains rejected. The improvement is an honest account of the outcome. Because two parts changed together, this example does not isolate a prompt effect.

The pending recording also retains wording that needs review. A correct structured status does not settle whether the message to the user is clear. Keeping that case visible makes the next engineering task more specific.

For your own comparison, retain the agent versions, test inputs, actual outputs and evaluation rules. Inspect regressions as well as improvements. Saved scores are useful only when you can trace them back to the evidence and understand what was left unmeasured.

Bring the result back to production

After a reviewed change ships, inspect new runs for the targeted failure and sample runs that were not flagged. Keep release identities and the relevant evidence so you can connect a result to the version that produced it. A merged pull request alone does not tell you which agent version handled a request.

The next issue adds a case to the regression set. A correction improves the evaluation policy. A failed repair gives you evidence for a different change. This is the feedback loop behind the new Sentrial homepage.

Where model training fits

Improving a signal classifier and training a customer’s agent are different tasks. A specialized classifier should be evaluated on independently reviewed, held-out examples, including healthy runs and uncertain outcomes. Training also requires permission to use the data and applicable model outputs. Compare false alerts, missed failures, abstentions, latency and serving cost before promoting a candidate.

Sentrial has not established an independently validated trained classifier or automatic customer-agent weight updates. The workflow available today centers on evidence, review, tests and controlled changes.

Start with the recorded insurance example, or run the published Node setup wizard from your agent project:

npx --yes --package @sentrial/sdk@0.8.1 sentrial setup

The wizard checks workspace access and prepares an integration example. Run the example, then verify delivery using the quickstart. Capture and signal evaluation need their own configuration; completing installation alone does not evaluate every run.