WHITEBOXX / GLOSSARY
Agent observability: seeing
what agents do in production.
AI agent observability is the ability to see what an agent does in production: each step it takes, each tool it calls, the tokens and time it uses, its errors and the quality of its results. Engineers use it to find faults, control costs and spot changes in behaviour.
What is agent observability?
Traditional monitoring tells you whether a service is up and how fast it responds. Agents need more, because the same request can take a different path each time. Observability for agents records a trace of every run: the model calls, tool calls and intermediate results, in order.
OpenTelemetry, the open standard for traces and metrics formed in 2019, publishes conventions for generative AI so that model calls and agent steps can be recorded in a common format.
SIGNALS
What to track.
| Signal | What it tells you |
|---|---|
| Traces | The sequence of steps, model calls and tool calls in each run |
| Tokens and cost | What each run and each step costs |
| Latency | Where the time goes |
| Errors and retries | Which tools or steps fail, and how often |
| Quality scores | Whether results still pass your evaluation tests |
Is observability the same as an audit trail?
No. Observability serves engineers who need to fix and improve the system. An audit trail serves reviewers who need to check a result: which data it used, which steps produced it and who approved it. The two can share records, but they answer different questions and are often kept for different lengths of time.
How Evidence OS records runs.
Evidence OS records each workflow run with its inputs, step progress, sources, calculations and deliverables, and the limits each agent ran with.
See the audit trail