WHITEBOXX / GLOSSARY

Agent observability: seeing
what agents do in production.

AI agent observability is the ability to see what an agent does in production: each step it takes, each tool it calls, the tokens and time it uses, its errors and the quality of its results. Engineers use it to find faults, control costs and spot changes in behaviour.

By Gautam Mehra · Published · Updated

What is agent observability?

Traditional monitoring tells you whether a service is up and how fast it responds. Agents need more, because the same request can take a different path each time. Observability for agents records a trace of every run: the model calls, tool calls and intermediate results, in order.

OpenTelemetry, the open standard for traces and metrics formed in 2019, publishes conventions for generative AI so that model calls and agent steps can be recorded in a common format.

SIGNALS

What to track.

SIGNALS
SignalWhat it tells you
TracesThe sequence of steps, model calls and tool calls in each run
Tokens and costWhat each run and each step costs
LatencyWhere the time goes
Errors and retriesWhich tools or steps fail, and how often
Quality scoresWhether results still pass your evaluation tests

Is observability the same as an audit trail?

No. Observability serves engineers who need to fix and improve the system. An audit trail serves reviewers who need to check a result: which data it used, which steps produced it and who approved it. The two can share records, but they answer different questions and are often kept for different lengths of time.

How Evidence OS records runs.

Evidence OS records each workflow run with its inputs, step progress, sources, calculations and deliverables, and the limits each agent ran with.

See the audit trail