How to detect false completion in AI agents

False completion happens when an agent says an action is finished but the available evidence contradicts that claim. Detecting it requires comparing the statement with the outcome of the same action. A successful model response, a tool invocation, or an HTTP success status alone does not prove that the user’s task was completed.

The difficult cases are usually ordinary: work accepted into a queue, a partial update, a timeout after a write, or a successful operation on the wrong object. This guide presents a proposed application design using synthetic examples. It is not a claim about a detector’s measured accuracy.

Define completion at the business boundary

Suppose a customer asks an agent to cancel order A17. The cancellation API returns an operation identifier immediately. A worker later decides whether the order can be cancelled. For this workflow, “request accepted” and “order cancelled” are different facts.

Write the completion contract before writing an evaluator. It should identify the action, its target, the authority that determines the outcome, and the terminal states. A tool response may be authoritative for a synchronous write; an asynchronous job may require a later status event. The correct boundary depends on the application.

Also distinguish the scope of the claim. “I requested cancellation” requires evidence of submission. “Your order is cancelled” requires evidence of cancellation. The evaluator should not demand the same proof for both statements.

Keep four states visible

Evidence state	What it establishes	Reasonable communication
Completed	The authoritative result confirms the intended action on the intended object.	Confirm the specific completed action.
Pending	The operation exists but has no terminal result yet.	Explain that processing is still underway.
Failed	The operation reached an explicit failure state.	Describe the failure and any supported next step.
Unknown	The available evidence cannot establish the outcome.	Say what is unverified and how it can be checked.

Unknown is essential. If a trace exporter drops the final result, you cannot safely infer that the underlying action failed. You can still identify an unsupported completion claim, but that is different from proving the action did not happen. Preserve that distinction in your labels and alerts.

Join the claim to the right action

Record an operation identifier, target object identifier, action type, attempt identifier, relevant timestamps, and final status. These are suggested application fields, not a universal telemetry standard. Avoid putting private customer content into identifiers or duplicating sensitive payloads unnecessarily.

A session can contain several similar actions. If order A17 fails to cancel and order A18 cancels successfully, the second result does not justify a completion claim about A17. Likewise, a successful read of the order is not proof of a successful mutation. Match both identity and operation semantics.

Retries complicate that join. A timed-out attempt may eventually commit; another may be rejected as a duplicate. Use the application’s operation or idempotency semantics to resolve the business outcome instead of selecting whichever span ended last.

Evaluate claims after the relevant evidence window

For asynchronous operations, decide when an assessment runs and what happens when new evidence arrives. An early assessment can say “pending” or “unsupported so far.” A later terminal result may justify revising the action state. Keep the assessment timestamp so reviewers know what evidence was available at the time.

Do not silently rewrite the historical meaning of the user-facing statement. If the agent promised completion before it was established, a later success may resolve the task while leaving a communication error worth fixing. Track operational outcome and claim accuracy separately.

Build a small, adversarial regression set

Start with paired examples: accepted then completed, accepted then rejected, timeout then committed, timeout with no later evidence, wrong-object success, partial completion, and an honest pending response. For every case, specify the expected action state and whether the final statement is supported.

Ask reviewers to cite the decisive event. If they disagree, inspect the contract before changing the model. Ambiguous application semantics cannot be repaired by a more confident judge.

For instrumentation structure, read connecting agent traces to business outcomes. To start exploring Sentrial, use the quickstart and validate that your own completion evidence reaches the investigation workflow.