> ## Documentation Index
> Fetch the complete documentation index at: https://docs.flowx.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Traces

> The chain-of-calls view for a single run — every chain, tool call, and event with timings and payloads.

Traces show the hierarchy of work inside one run. Use this view when you need to understand what happened end-to-end — which sub-chain decided to call which tool, how long each step took, and where the request went wrong.

***

## Span hierarchy

A trace is a tree of spans. Each span is one event the SDK reported:

```mermaid theme={"system"}
flowchart TD
    Root[agent_start: claims-approval]
    Root --> C1[chain_start: extract-claim-data]
    C1 --> T1[tool_start: lookup_policy_holder]
    T1 --> T1E[tool_end]
    C1 --> C1E[chain_end]
    Root --> C2[chain_start: assess-risk]
    C2 --> L1[llm_call: gpt-4]
    C2 --> C2E[chain_end]
    Root --> RootE[agent_end]
```

Spans are nested by `parent_run_id`. The first span (root) is the agent call itself. Each chain or tool nested inside gets its own span. Leaf spans are the LLM calls or tool invocations.

***

## What you see per span

| Field              | Meaning                                                                             |
| ------------------ | ----------------------------------------------------------------------------------- |
| **Name**           | Decorator string from `@chain("...")`, `@tool("...")`, or the LangChain class name. |
| **Type**           | `agent`, `chain`, `tool`, `llm`, or a custom event.                                 |
| **Duration**       | Wall-clock milliseconds between `*_start` and `*_end`.                              |
| **Status**         | `success`, `error`, or `in_progress`.                                               |
| **Input / output** | Raw payloads, rendered as text or JSON.                                             |
| **Tokens**         | Per-LLM-call token usage when the provider reports it.                              |
| **Cost**           | Derived from token usage × the model's per-1k-token rate.                           |

***

## Reading a trace

<Steps>
  <Step title="Open the run">
    From [LLM Calls](./llm-calls) or [Threads](./threads), click a row to open the trace detail.
  </Step>

  <Step title="Scan the tree for red">
    Failed spans are highlighted. Their parent chain typically captures the error message in the output.
  </Step>

  <Step title="Inspect the slow spans">
    Sort or eyeball the duration column. A single tool call dominating the duration usually points at the bottleneck.
  </Step>

  <Step title="Open the LLM call">
    Click any `llm` span to see the prompt, system message, and completion alongside token usage.
  </Step>
</Steps>

***

## Comparing two runs

Observatory ships a comparison endpoint that surfaces side-by-side diffs of two runs. From the trace view, click **Compare** and pick a second run. Useful for:

* Investigating a regression after a prompt change
* Comparing the same input across two model versions
* Diffing a passing dataset run against a failing one

```http theme={"system"}
POST /api/run/compare
{ "run_id_a": "...", "run_id_b": "..." }
```

***

## Saved runs

A run is marked **saved** when an event with name `saved` is emitted for it. Saved runs persist beyond retention windows and surface in dataset workflows.

```python theme={"system"}
from flowxobservatory import track_event

track_event("saved", run_id=ctx.run_id)
```

See [Event reporting](../sdk/event-reporting#custom-events) for the full list of custom events.

***

## Related resources

<CardGroup cols={2}>
  <Card title="LLM Calls" icon="bolt" href="./llm-calls">
    The table view that links into trace detail.
  </Card>

  <Card title="Threads" icon="comments" href="./threads">
    Multi-turn views built on the same trace data.
  </Card>
</CardGroup>