AI observability
YZ gözlemlenebilirliği
D4
AI observability is the use of operational signals and context to investigate how an AI-enabled system behaves in use and to investigate the causes of problems.
Review status: 2026-12-04
Technical explanation
Useful evidence can connect requests, model and tool steps, outcomes, errors, versions, and relevant context so operators can ask why a behavior occurred and follow it through a workflow.
Conceptual boundaries
Observability is broader than collecting logs and different from an evaluation: logs are one signal, while evaluation tests defined quality questions; neither alone proves real-world correctness or safety.
Provider-neutral example
An operator investigates a bad answer by joining a request trace with retrieved-document identifiers, tool results, model version, latency, error events, and a reviewed outcome label.
Limitations
Signals can be incomplete, sampled, privacy-sensitive, or poorly correlated; instrumentation cannot substitute for evaluation, access control, or accountable human judgment.
Related concepts
Atomic claims and evidence
1.1OpenTelemetry, Observability primer
- Source
- OpenTelemetry, Observability primer
- Source role
- Authoritative source
- Exact locator
- What is Observability?: system from outside, novel problems, and why
- Supported claim
- OpenTelemetry defines observability as understanding a system from the outside, including troubleshooting novel problems and asking why behavior occurs.
- Last verification
- Review due
- Scope limitation
- This is OpenTelemetry's general systems-observability concept; it does not define a complete AI-specific measurement program.
2.1OpenTelemetry Semantic Conventions, Generative AI registry
- Source
- OpenTelemetry Semantic Conventions, Generative AI registry
- Source role
- Authoritative source
- Exact locator
- GenAI registry: model, agent version, tool definitions, tool type, and operation context
- Supported claim
- OpenTelemetry's Generative AI semantic-conventions registry includes attributes for model, agent version, tool definitions, tool type, and operation context.
- Last verification
- Review due
- Scope limitation
- A semantic vocabulary enables more consistent capture and correlation; it does not establish that an implementation records sufficient AI context or reaches correct conclusions.
3.1NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile
- Source
- NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile
- Source role
- Authoritative source
- Exact locator
- Printed p. 27: known ground truth and varied evaluation methods
- Supported claim
- NIST recommends varied evaluation methods and known ground truth for generative-AI information integrity, so operational observability should not be treated as a replacement for evaluation.
- Last verification
- Review due
- Scope limitation
- The conclusion about separation is an inference from the distinct purposes of operational signals and evaluation methods, not a claim that either is unnecessary.