Observability / FIELD NOTES
How to investigate an AI agent across multiple systems
Build a bounded incident timeline from agent runs, tool requests and destination records while preserving gaps and competing explanations.
Start with one disputed outcome, follow stable identifiers across boundaries, and separate observed events from the explanation you are testing.
An agent investigation often begins with a symptom: a customer received two messages, a record changed unexpectedly or an operational task appears to have completed without its expected effect. The evidence may be split across the conversation, an agent runtime, a queue, an API and the destination's own history.
The practical challenge is to connect these observations without treating proximity as proof. Two events occurring within the same minute may be related, but their timestamps alone do not establish that one caused the other. An investigation needs a bounded question and a visible distinction between observations and hypotheses.
Define the disputed outcome
Start by writing a question that can be answered. For a hypothetical duplicate-message incident: “Which operations caused two messages to be accepted for recipient R-22 between 09:00 and 09:10 UTC?” This is more useful than “Why did the agent behave badly?” It identifies a destination, an object and an interval.
Record the original report without silently converting it into a verified fact. A screenshot may show two messages, while the destination's event history may reveal whether they came from the same workflow. Both belong in the case, with their sources identified.
Choose an initial interval wide enough to include submission and delayed processing, then expand it only when new evidence justifies doing so. This keeps the investigation proportionate and makes the final scope explainable.
Follow identifiers through each boundary
OpenTelemetry context propagation carries trace and span identifiers between services so related operations can be correlated. It provides a standard mechanism for connecting instrumentation across boundaries. OpenTelemetry's context propagation guide describes that mechanism.
For our hypothetical incident, collect the task identifier, run identifier, tool request identifier, queue job identifier and destination message identifier where they actually exist. Preserve the mapping returned at each boundary. A destination identifier copied from a successful API response is more useful than an inferred match based only on message text.
Do not assume every downstream service retains the trace context. If a boundary drops it, record the gap and look for another documented reference. When no reliable join exists, mark the association as tentative rather than silently merging two records.
Build a timeline with provenance
A useful working timeline records what happened, when the source says it happened, when the investigator received it and where the evidence came from. Keep the original event identifier and timezone. If clocks disagree, document the disagreement instead of adjusting timestamps until the narrative looks neat.
For the duplicate-message example, the first pass might show one agent instruction, two tool attempts and two destination acknowledgements. That supports an observation about duplication. It does not yet establish whether the second attempt followed a timeout, a user correction or a separate instruction.
Add the relevant inputs and tool results next. Look for the first point where the recorded behavior differs from the intended workflow. This is often more productive than beginning with the final generated response.
Keep asynchronous work visible
Queues and background tasks can outlive the caller that started them. OpenTelemetry supports span links for associating operations, including work that begins asynchronously in another trace. Its span-link documentation explains that relationship.
Our recommendation is to retain both the submission and processing records, even if the user-facing run has already ended. A parent task marked complete should not erase the fact that a destination job remained pending. Likewise, a later cancellation or reversal should be attached as a later event, not used to rewrite the original observation.
Test competing explanations
Write down at least the plausible explanations supported by the evidence. In this example, the second message could come from a retry after an ambiguous timeout, a duplicate queue delivery or an independent request. For each explanation, identify one observation that would distinguish it from the others.
Then pursue that observation. Check the retry record, queue delivery identifier or original instruction. Avoid changing a prompt merely because it is the easiest editable component. If the destination received the same operation twice, the repair may belong in request handling or deduplication instead.
Close with scope and remaining uncertainty
An investigation report should state the verified sequence, the cause supported by the collected evidence, the bounded repair and the checks that challenged it. Include unresolved gaps. If destination history was unavailable for part of the interval, say so explicitly.
After the repair, test the affected journey and an adjacent working journey. In the duplicate-message case, include a delayed response, an explicit user correction and a new enquiry. Verify destination results when possible. The investigation is useful when another engineer can follow the same references and understand both the conclusion and its limits.
Sources and further reading
Primary sources checked on 16 November 2026. Examples in this article are illustrative unless stated otherwise.