Observability¶

AI Guardian exports OpenTelemetry (OTEL) traces for both SDK agent runs and interactive IDE sessions. Traces follow the OpenTelemetry GenAI semantic conventions and work with any OTLP-compatible backend: Grafana Tempo, Jaeger, Datadog, Splunk, Honeycomb.

Quick Start¶

1. Start a collector¶

The Grafana LGTM stack (Loki, Grafana, Tempo, Mimir) runs everything in one container:

docker run -d --name lgtm \
  -p 3000:3000 \
  -p 4317:4317 \
  -p 4318:4318 \
  grafana/otel-lgtm

3000 — Grafana UI
4317 — OTLP gRPC (not used by ai-guardian)
4318 — OTLP HTTP (ai-guardian sends here)

2. Enable OTEL export¶

Add to your ai-guardian.json (or .ai-guardian/ai-guardian.json):

{
  "otel": {
    "enabled": true,
    "endpoint": "http://localhost:4318"
  }
}

3. View traces¶

Open Grafana at http://localhost:3000, navigate to Explore > Tempo, and query:

{ resource.service.name = "ai-guardian" }

Configuration¶

OTEL configuration lives in the top-level otel section of ai-guardian.json.

{
  "otel": {
    "enabled": true,
    "endpoint": "http://localhost:4318",
    "service_name": "ai-guardian",
    "export_format": "otlp-json",
    "headers": {
      "Authorization": "Bearer <token>"
    },
    "resource_attributes": {
      "team.name": "security",
      "deployment.environment": "dev"
    }
  }
}

Key	Type	Default	Description
`enabled`	bool	`false`	Enable OTEL span export
`endpoint`	string	`http://localhost:4318`	OTLP HTTP collector URL
`service_name`	string	`ai-guardian`	`service.name` resource attribute
`export_format`	string	`otlp-json`	`otlp-json` or `otlp-proto` (proto requires `pip install ai-guardian[otel]`)
`headers`	object	`{}`	HTTP headers sent with each export request
`resource_attributes`	object	`{}`	Static key-value pairs added as OTEL resource attributes on every span

Environment Variable Overrides¶

Env Var	Overrides	Format
`OTEL_EXPORTER_OTLP_ENDPOINT`	`endpoint`	URL
`OTEL_SERVICE_NAME`	`service_name`	string
`OTEL_EXPORTER_OTLP_HEADERS`	`headers` (config takes precedence)	`key1=val1,key2=val2`

Unified Policy Decisions¶

Violation spans include safe ai_guardian.policy.* attributes derived from the versioned policy decision schema. The attributes include the event, decision, reason, severity, confidence, policy version, source, agent, repository, correlation ID, and latency. Raw commands, matched text, snippets, and other scanner payloads are not copied into these attributes.

The same decision metadata is embedded in violations.jsonl, returned by the daemon REST violation and check endpoints, and exposed in SARIF result properties. Existing fields and OTEL span names remain compatible with older consumers.

What Gets Exported¶

SDK Agent Runs (GuardedAgent)¶

Full agent conversation traces with per-turn detail. Exported in two ways:

Live export — each turn is sent to the collector as it completes (via OtelSpanEmitter)
File export — trace JSON files written to ~/.local/state/ai-guardian/sdk/traces/, exportable via CLI

Span Hierarchy¶

gen_ai.agent (root)
└── gen_ai.turn (one per conversation turn)
    ├── gen_ai.chat          — LLM response with token usage
    ├── tool:{name}          — tool call (e.g., tool:bash)
    ├── tool_result:{name}   — tool result
    ├── gen_ai.security_scan — security scan result
    └── gen_ai.compaction    — context compaction event

Root Span Attributes (`gen_ai.agent`)¶

Attribute	Type	Description
`gen_ai.system`	string	Always `"anthropic"`
`gen_ai.agent.name`	string	Agent name
`gen_ai.request.model`	string	Model ID (e.g., `claude-sonnet-4-20250514`)
`gen_ai.request.max_tokens`	int	Max output tokens
`gen_ai.agent.stop_reason`	string	`end_turn`, `error`, etc.
`gen_ai.agent.duration_ms`	int	Total run duration in milliseconds
`gen_ai.agent.hostname`	string	Machine hostname
`ai_guardian.session_id`	string	Session identifier (if available)
`ai_guardian.project.name`	string	`org/repo` from git remote, or directory basename
`ai_guardian.span_type`	string	`"agent_run"` (on live export root spans)

Token usage totals are not on the root span. Per-turn usage lives on gen_ai.chat child spans — Grafana can sum them with sum(gen_ai.usage.input_tokens).

Chat Span Attributes (`gen_ai.chat`)¶

Attribute	Type	Description
`gen_ai.usage.input_tokens`	int	Input tokens for this turn
`gen_ai.usage.output_tokens`	int	Output tokens for this turn
`gen_ai.usage.cache_read_input_tokens`	int	Cache read tokens
`gen_ai.usage.cache_creation_input_tokens`	int	Cache creation tokens
`gen_ai.response.finish_reasons`	array	Stop reason (`end_turn`, `tool_use`)
`gen_ai.chat.latency_ms`	int	LLM response latency
`gen_ai.response.text_length`	int	Response text length in characters
`gen_ai.response.tool_call_count`	int	Number of tool calls in this response

Turn Span Attributes (`gen_ai.turn`)¶

Attribute	Type	Description
`gen_ai.turn.number`	int	Turn index (0-based)
`gen_ai.turn.messages_count`	int	Number of messages in context
`gen_ai.turn.messages_count_growth`	int	Messages added since previous turn
`gen_ai.turn.compacted`	bool	Whether context was compacted before this turn
`gen_ai.turn.duration_ms`	int	Turn duration

Tool Span Attributes¶

Attribute	Type	Span
`tool.name`	string	Both `tool:{name}` and `tool_result:{name}`
`tool.input`	string	`tool:{name}` — JSON input (truncated to 1024 chars)
`tool.output_bytes`	int	`tool_result:{name}` — output size
`tool.output_truncated`	bool	`tool_result:{name}` — whether output was truncated
`tool.latency_ms`	int	`tool_result:{name}` — execution time

Interactive Sessions (Hooks)¶

Session-level activity summary exported on SessionEnd. Works in both daemon and direct mode — violations are read from violations.jsonl at flush time, so no in-memory state is required between hook calls.

Span Hierarchy¶

ai_guardian.session (root)
├── ai_guardian.violation  — one per non-blocking violation (warn mode)
└── ai_guardian.block      — one per blocked tool call

Session Root Attributes (`ai_guardian.session`)¶

Attribute	Type	Description
`ai_guardian.session_id`	string	IDE session identifier
`ai_guardian.session_sequence`	int	Session open count (increments on reopen)
`ai_guardian.project.name`	string	`org/repo` from git remote, or directory basename
`ai_guardian.adapter`	string	IDE adapter name (e.g., `Claude Code`, `Gemini CLI`)
`ai_guardian.violation_count`	int	Total non-blocking violations in session
`ai_guardian.block_count`	int	Total blocked tool calls
`ai_guardian.span_type`	string	Always `"session"`
`gen_ai.usage.input_tokens`	int	Total input tokens (from transcript, if available)
`gen_ai.usage.output_tokens`	int	Total output tokens
`gen_ai.usage.cache_read_input_tokens`	int	Total cache read tokens
`gen_ai.usage.cache_creation_input_tokens`	int	Total cache creation tokens

Violation Span Attributes (`ai_guardian.violation`)¶

Attribute	Type	Description
`ai_guardian.violation_type`	string	e.g., `secret_detected`, `prompt_injection`
`ai_guardian.severity`	string	`warning`, `critical`
`tool.name`	string	Tool that triggered the violation
`ai_guardian.violation_id`	string	Unique violation identifier
`ai_guardian.scanner`	string	Scanner that detected it

Block Span Attributes (`ai_guardian.block`)¶

Attribute	Type	Description
`tool.name`	string	Tool that was blocked
`ai_guardian.reason`	string	Why the tool call was blocked
`ai_guardian.scanner`	string	Scanner that blocked it

SDK Dynamic Metadata¶

GuardedAgent accepts an otel_metadata_fn callback to add custom attributes to every span:

from ai_guardian.integrations.anthropic import GuardedAgent

def metadata_fn(agent_name: str, ctx: dict) -> dict:
    """Add custom OTEL attributes.

    ctx contains: model, turn, usage, and stop_reason (on run complete).
    Return a dict of key-value pairs.
    """
    return {
        "case.id": "AAP-85065",
        "pipeline.stage": "triage",
    }

agent = GuardedAgent(
    name="triage-agent",
    model="claude-sonnet-4-20250514",
    otel_metadata_fn=metadata_fn,
)

The callback is called: - On each turn completion (with turn, model, usage) - On run completion (adds stop_reason to the context)

Returned key-value pairs are added as OTEL attributes on the turn or root span.

CLI Export¶

Export saved trace files to OTLP format or directly to a collector.

Export a single trace¶

# To stdout (OTLP JSON)
ai-guardian trace export trace.json

# To file
ai-guardian trace export trace.json -o trace.otlp.json

# To collector
ai-guardian trace export trace.json --endpoint http://localhost:4318

# With auth header
ai-guardian trace export trace.json \
  --endpoint http://collector:4318 \
  --header 'Authorization=Bearer tok123'

# Protobuf format (requires pip install ai-guardian[otel])
ai-guardian trace export trace.json --format otlp-proto -o trace.pb

Export all traces in a directory¶

# Export SDK trace directory (auto-detected)
ai-guardian trace export-dir

# Export specific directory
ai-guardian trace export-dir ~/.local/state/ai-guardian/sdk/traces/ -o ./exported/

# Custom service name
ai-guardian trace export-dir --service-name my-pipeline -o ./exported/

Trace files are stored in ~/.local/state/ai-guardian/sdk/traces/ by default (XDG state directory).

Grafana Setup¶

Data Source Configuration¶

After starting the LGTM stack:

Open Grafana at http://localhost:3000 (default credentials: admin/admin)
Go to Connections > Data sources > Tempo (pre-configured in LGTM)
Navigate to Explore > Tempo

Useful TraceQL Queries¶

# All traces from ai-guardian
{ resource.service.name = "ai-guardian" }

# Filter by agent name
{ span.gen_ai.agent.name = "triage-agent" }

# Filter by project
{ span.ai_guardian.project.name = "RedHatProductSecurity/ai-guardian" }

# All activity for an org
{ span.ai_guardian.project.name =~ "RedHatProductSecurity/.*" }

# Sessions with violations
{ span.ai_guardian.violation_count > 0 }

# Find specific violation types
{ name = "ai_guardian.violation" && span.ai_guardian.violation_type = "secret_detected" }

# Blocked tool calls
{ name = "ai_guardian.block" }

# Token-heavy turns (> 10k input tokens)
{ name = "gen_ai.chat" && span.gen_ai.usage.input_tokens > 10000 }

# Compaction events
{ name = "gen_ai.compaction" }

# Filter by team (via resource_attributes)
{ resource.team.name = "security" }

# Errors only
{ name = "gen_ai.agent" && status = error }

Trace Viewer (Console)¶

AI Guardian includes a built-in trace viewer accessible from both the TUI and web console.

Web Console¶

Navigate to http://{daemon-host}:{port}/traces to browse SDK agent traces. The page shows a list of trace files with: - Agent name, model, duration - Turn count and token usage - Explicit unavailable token usage when the agent does not expose usage data - A Send to Collector button to push traces to the configured OTEL endpoint

TUI Console¶

Run ai-guardian console --web and navigate to the SDK Traces panel for the same functionality in a terminal interface.

Anthropic Console vs ai-guardian OTEL¶

Aspect	Anthropic Console	ai-guardian OTEL
Level	API-level (per-request)	Agent-level (full conversation)
Scope	Single API call	Multi-turn agent run with tool calls
Security	Not included	Violations, blocks, scan results
Custom metadata	Not supported	`otel_metadata_fn` callback
Backend	Anthropic dashboard	Any OTLP backend (Grafana, Jaeger, etc.)
Interactive sessions	Not supported	Hook session activity
Self-hosted	No	Yes

Use Anthropic Console for API debugging and rate limit monitoring. Use ai-guardian OTEL for agent-level observability, security auditing, and cross-project analysis.

Troubleshooting¶

Traces not showing in Grafana¶

Check time range — Grafana defaults to "Last 1 hour". Widen the range or use "Last 15 minutes" for recent traces.
Check endpoint — Verify the collector is running: curl -s http://localhost:4318/v1/traces -X POST -H 'Content-Type: application/json' -d '{}'
Check config — Run ai-guardian config show and look for the otel section.

Collector not accessible¶

OTEL export is fail-open: if the collector is unreachable, the agent continues normally. Failures are logged at DEBUG level. Check logs with:

AI_GUARDIAN_LOG_LEVEL=DEBUG ai-guardian trace export trace.json --endpoint http://localhost:4318

Hook sessions not exporting¶

Verify OTEL is enabled in config: ai-guardian config show | grep -A5 otel
If using the daemon, restart after config changes: ai-guardian daemon restart
Hook session traces are flushed on SessionEnd — end the IDE session to trigger export.
Direct mode (no daemon) also exports — violations are read from violations.jsonl at session end.

Protobuf format errors¶

The otlp-proto format requires additional dependencies:

pip install ai-guardian[otel]

This installs opentelemetry-proto and protobuf. The default otlp-json format requires no extra dependencies.

Observability¶

Quick Start¶

1. Start a collector¶

2. Enable OTEL export¶

3. View traces¶

Configuration¶

Environment Variable Overrides¶

Unified Policy Decisions¶

What Gets Exported¶

SDK Agent Runs (GuardedAgent)¶

Span Hierarchy¶

Root Span Attributes (gen_ai.agent)¶

Chat Span Attributes (gen_ai.chat)¶

Turn Span Attributes (gen_ai.turn)¶

Tool Span Attributes¶

Interactive Sessions (Hooks)¶

Span Hierarchy¶

Session Root Attributes (ai_guardian.session)¶

Violation Span Attributes (ai_guardian.violation)¶

Block Span Attributes (ai_guardian.block)¶

SDK Dynamic Metadata¶

CLI Export¶

Export a single trace¶

Export all traces in a directory¶

Grafana Setup¶

Data Source Configuration¶

Useful TraceQL Queries¶

Trace Viewer (Console)¶

Web Console¶

TUI Console¶

Anthropic Console vs ai-guardian OTEL¶

Troubleshooting¶

Traces not showing in Grafana¶

Collector not accessible¶

Hook sessions not exporting¶

Protobuf format errors¶

Root Span Attributes (`gen_ai.agent`)¶

Chat Span Attributes (`gen_ai.chat`)¶

Turn Span Attributes (`gen_ai.turn`)¶

Session Root Attributes (`ai_guardian.session`)¶

Violation Span Attributes (`ai_guardian.violation`)¶

Block Span Attributes (`ai_guardian.block`)¶