> ## Documentation Index
> Fetch the complete documentation index at: https://docs.flowx.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Traces

> The chain-of-calls view for a single run — every chain, tool call, and event with timings and payloads.

Traces show the hierarchy of work inside one run. Use this view when you need to understand what happened end-to-end — which sub-chain decided to call which tool, how long each step took, and where the request went wrong.

***

## Span hierarchy

A trace is a tree of spans. Each span is one event the SDK reported:

```mermaid theme={"dark"}
flowchart TD
    Root[agent_start: claims-approval]
    Root --> C1[chain_start: extract-claim-data]
    C1 --> T1[tool_start: lookup_policy_holder]
    T1 --> T1E[tool_end]
    C1 --> C1E[chain_end]
    Root --> C2[chain_start: assess-risk]
    C2 --> L1[llm_call: gpt-4]
    C2 --> C2E[chain_end]
    Root --> RootE[agent_end]
```

Spans are nested by `parent_run_id`. The first span (root) is the agent call itself. Each chain or tool nested inside gets its own span. Leaf spans are the LLM calls or tool invocations.

***

## What you see per span

| Field | Meaning |
| - | - |
| **Name** | Decorator string from `@chain("...")`, `@tool("...")`, or the LangChain class name. |
| **Type** | `agent`, `chain`, `tool`, `llm`, or a custom event. |
| **Duration** | Wall-clock milliseconds between `*_start` and `*_end`. |
| **Status** | `success`, `error`, or `in_progress`. |
| **Input / output** | Raw payloads, rendered as text or JSON. |
| **Tokens** | Per-LLM-call token usage when the provider reports it. |
| **Cost** | Derived from token usage × the model's per-1k-token rate. |

***

## Reading a trace

**1. Open the run**

From [LLM Calls](https://docs.flowx.ai/6.x/observatory/observability/llm-calls.md) or [Threads](https://docs.flowx.ai/6.x/observatory/observability/threads.md), click a row to open the trace detail.

**2. Scan the tree for red**

Failed spans are highlighted. Their parent chain typically captures the error message in the output.

**3. Inspect the slow spans**

Sort or eyeball the duration column. A single tool call dominating the duration usually points at the bottleneck.

**4. Open the LLM call**

Click any `llm` span to see the prompt, system message, and completion alongside token usage.

***

## Comparing two runs

Observatory ships a comparison endpoint that surfaces side-by-side diffs of two runs. From the trace view, click **Compare** and pick a second run. Useful for:

* Investigating a regression after a prompt change
* Comparing the same input across two model versions
* Diffing a passing dataset run against a failing one

```http theme={"dark"}
POST /api/run/compare
{ "run_id_a": "...", "run_id_b": "..." }
```

***

## Saved runs

A run is marked **saved** when an event with name `saved` is emitted for it. Saved runs persist beyond retention windows and surface in dataset workflows.

```python theme={"dark"}
from flowxobservatory import track_event

track_event("saved", run_id=ctx.run_id)
```

See [Event reporting](https://docs.flowx.ai/6.x/observatory/sdk/event-reporting.md#custom-events) for the full list of custom events.

***

## Related resources

> [card] [LLM Calls](https://docs.flowx.ai/6.x/observatory/observability/llm-calls.md)
>
> The table view that links into trace detail.

> [card] [Threads](https://docs.flowx.ai/6.x/observatory/observability/threads.md)
>
> Multi-turn views built on the same trace data.


## Related topics

- [Observability with OpenTelemetry](https://docs.flowx.ai/6.x/docs/platform-deep-dive/integrations/open-telemetry.md)
- [Threads](https://docs.flowx.ai/6.x/observatory/observability/threads.md)
- [Open Telemetry default properties](https://docs.flowx.ai/6.x/setup-guides/ot-default-properties.md)
- [Getting started](https://docs.flowx.ai/6.x/observatory/getting-started.md)
- [LLM Calls](https://docs.flowx.ai/6.x/observatory/observability/llm-calls.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.