> ## Documentation Index
> Fetch the complete documentation index at: https://docs.flowx.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluations

> Test AI nodes with test cases: check outputs with asserts, score quality with LLM evaluators, and run experiments across tagged cases.

> [!NOTE]
>
> Write test cases for an AI node in a workflow, check its output with deterministic asserts, score its quality with LLM evaluators, and run an experiment over a tagged set of cases. Judge scoring runs asynchronously in the `evals-judge` service.

## Overview

An *evaluation* (eval) checks how an AI node behaves on a known input. Use evals to:

* **Tune prompts and retrieval.** Change a system prompt or a knowledge base setting, rerun the same test cases, and see whether the results improved.
* **Guard against regressions.** Before you commit a new agent version, run your set of hard past inputs and compare the results with the previous run.
* **Pick a model.** Run the same test cases against two LLMs and compare the verdicts and quality scores.

Evals are workflow-centric. Open a workflow and select the flask icon (**Test Cases**) in the left sidebar. The page has three tabs: **Evaluators**, **Cases**, and **Experiments**.

![The Test Cases page of a workflow with the Evaluators tab open, listing the General Quality evaluators Answer Relevance, Conciseness, Correctness, and Hallucination, each with Numeric scoring and a 70% pass threshold](https://s3.eu-west-1.amazonaws.com/docx.flowx.ai/evals-evaluators-tab.png)

***

## Core concepts

| Concept | What it is |
| - | - |
| **Test case** | A JSON input for one AI node (the case's **target**), plus the checks its output must pass. Grouped with **tags**. |
| **Assert** | A deterministic check on the node output, such as **equals** an expected value. Gives a pass or fail verdict. |
| **Evaluator** | An LLM judge that scores the quality of the output, such as correctness or groundedness, against a pass threshold. |
| **Experiment** | A run of a selected set of test cases. Freezes the configuration of each target node and records a result for every case. |

A test case needs at least one check: an assert, an evaluator, or both.

***

## Get started

**1. Open the Test Cases page**

In a workflow that contains an AI node, select the flask icon (**Test Cases**) in the left sidebar. You can also select the flask button on an AI node on the canvas, which opens the **Cases** tab filtered to that node.

**2. Create a test case**

On the **Cases** tab, click **New test case**. Enter a **Case name**, pick the **Target** node, and add **Tags** to group the case with others.

**3. Enter the input and add checks**

Enter the JSON **Input**. Under **Test case**, use **Add** to add an **Assert**, an **Eval**, or both.

**4. Run the case and save it**

Click **Run** to run the node on its own over the input. Saving keeps the output as the case output. Click **Create case**.

**5. Run an experiment**

On the **Experiments** tab, click **New experiment**, name the run, select the tags or test cases to include, and click **Run tests**.

**6. Read the results**

Open the experiment to see assert verdicts and evaluator scores per target, with a result for every case.

***

## Create test cases

A test case belongs to one workflow and tests one AI node. Every AI node type supports test cases except **Document Generation**, **Context Retrieval**, and **Deep Research**.

| Field | Required | Description |
| - | - | - |
| **Case name** | Yes | Identifies the case in lists and reports. Up to 200 characters. |
| **Target** | Yes | The AI node the case tests. Can't be changed after the case is created. |
| **Tags** | No | Labels that group cases. Experiments select test cases by tag. Type a new tag to create it. |
| **Input** | Yes | The JSON data the node runs on. Must be valid JSON. `null`, `false`, `0`, and empty collections are valid values. |

You can add test cases in three ways:

> [tab] Manually
>
> On the **Cases** tab, click **New test case** and fill in the fields. Under **Test case**, click **Run** to run the node on its own over the input, then add asserts and evaluators that are checked against that output.
>
> If you save without running, the case has no output yet. The asserts and evaluators are checked when the case runs in an experiment.

> [tab] From a workflow run
>
> Run the workflow from the editor. In the run console, select the flask button (**Add to a test case**) on the AI node entry. A new test case opens, prefilled with the input and output of that run. If the execution was already captured, the menu lists the test cases that contain it.
>
> Capture is available only for successful AI node executions in the editor console, not on a committed version or in a child run. It isn't available when the node had no output or its payload was too large to store inline.

> [tab] From the canvas
>
> Select the flask button on an AI node on the workflow canvas. The **Cases** tab opens, filtered to that node, so you can add or edit its test cases.

![The New test case dialog with Case name, Tags, and a Target set to an AI node, showing one equals assert that checks the whole node output against an expected value](https://s3.eu-west-1.amazonaws.com/docx.flowx.ai/evals-new-test-case-assert.png)

Use the **Cases** tab filters (**All Tags**, **All Targets**, and **Search by name**) to find cases. Each row shows the case target, its number of asserts and evaluators, and the result of its last run.

Test cases and custom evaluators travel with the workflow when you clone, export, import, or merge it. A clone or import doesn't add test cases to a workflow that already has some.

***

## Add asserts

An assert checks the node output against a value you expect and gives a verdict: met or not met. Asserts don't call an LLM.

| Assert | What it checks |
| - | - |
| **equals** | The output, or a value inside it, equals the expected value: a number, `true` or `false`, or text. |
| **equals as JSON** | The output equals the expected JSON. The number type matters, so `1` and `1.0` are different. |

Mark an assert **Soft** to report it without failing the case. A soft assert that isn't met shows in the report under **Soft not met**.

> [!NOTE]
>
> **The Code evaluators are now asserts.** The **Exact match** and **JSON match** evaluators from earlier versions are replaced by the **equals** and **equals as JSON** asserts. **Levenshtein** similarity has no replacement.

***

## Add evaluators

An evaluator is an LLM judge that scores the output from 0 to 1 against a pass threshold. In a test case, add one with **Add** → **Eval**, pick the evaluator, and optionally change the pass threshold for that case.

The **Evaluators** tab lists the evaluators available in the workflow: the built-in ones, grouped by category, and any you create. The built-in evaluators can't be edited or deleted.

### General quality

A judge LLM reads the node output, and the expected output when there is one, and scores it on a single rubric. All four use **Numeric** scoring with a default pass threshold of 70%.

| Evaluator | What it scores |
| - | - |
| **Correctness** | Whether the output contains the same key facts as the expected output. |
| **Conciseness** | Whether the output is appropriately brief. Penalizes padding and repetition. |
| **Hallucination** | Whether the output makes claims not supported by the provided context or the expected output. |
| **Answer Relevance** | Whether the output answers what the input asked, regardless of factual correctness. |

### RAG

Evaluators for retrieval-augmented generation, covering both retrieval and the answer. They apply to a Custom Agent node that uses a knowledge base. All three use **Numeric** scoring with a default pass threshold of 70%.

| Evaluator | What it scores |
| - | - |
| **RAG Groundedness** | Whether every claim in the output traces back to the retrieved context, rather than the model's prior knowledge. |
| **RAG Helpfulness** | How useful the answer is to the end user, considering tone, completeness, and actionability. |
| **RAG Retrieval Relevance** | How relevant the retrieved documents are to the question. A quality check for the retrieval step. |

RAG Groundedness and RAG Retrieval Relevance need retrieved context. Without it, the check errors instead of scoring.

### Custom

Write your own evaluator when none of the built-in rubrics measures what you care about. A custom evaluator is scored by the judge LLM following instructions you supply, and appears under the **Custom** group on the **Evaluators** tab, after the built-in ones.

Custom evaluators belong to **one workflow**. Click **New evaluator** on the **Evaluators** tab to create one.

| Field | Required | What it does |
| - | - | - |
| **Start from a template** | No | Prefills the form from an existing evaluator, instructions included, so you can adapt a built-in rubric instead of starting from a blank prompt. |
| **Name** | Yes | Identifies the evaluator on the **Evaluators** tab and in experiment reports. Must be unique in the workflow. |
| **Description** | No | Records what the evaluator measures, for whoever reads the list next. |
| **Instructions** | Yes | The prompt sent to the judge model for every test case output. Tell the judge what to look for and how to score it. |
| **Scoring** | Yes | **Numeric** for a score from 0 to 1, or **Binary** for pass or fail. Numeric is the default. |
| **Pass threshold** | Only when scoring is **Numeric** | The score a case must reach to pass. Must be between `0` and `1`. |

> [!NOTE]
>
> **Binary scoring has no threshold.** Switch **Scoring** to **Binary** and the **Pass threshold** field is turned off, since pass or fail is already the result.

Unlike the built-in evaluators, the ones you create can be edited and deleted. A deleted evaluator is no longer available for new experiments. Past experiment reports keep their results.

***

## Run an experiment

An experiment runs a selected set of test cases. Each case runs against its own target node, so one experiment can cover several AI nodes in the workflow. The report groups the results by target.

**1. Start a new experiment**

On the **Experiments** tab, click **New experiment** and enter a **Name** for the run.

**2. Select the test cases**

Select tags to include the test cases that carry them, then use **Edit selection** to add or remove individual cases. Cases whose target node was deleted are skipped.

**3. Run the tests**

Click **Run tests**. The configuration of each target node is frozen for this run, and the report fills in live as the checks complete.

![The New experiment dialog with a Name field, a Test cases section to select tags and edit the selection, and a Run tests button](https://s3.eu-west-1.amazonaws.com/docx.flowx.ai/evals-new-experiment.png)

An experiment doesn't start if no selected case has an enabled check, if every selected case targets a node that no longer exists, or if a case input isn't valid JSON.

Experiment status moves through:

| Status | Meaning |
| - | - |
| `STARTED` | The experiment is running. |
| `FINISHED` | Every case ran without errors. Individual cases can still have failed their checks. |
| `PARTIAL` | Some cases errored. The results of the other cases are available. |
| `FAILED` | Every case errored. |
| `CANCELLED` | Cancelled by a user while running. Cases still in progress are abandoned; results already recorded are kept. |

You can cancel an experiment only while it is running. Deleting an experiment removes its results and keeps the test cases.

***

## View results

Open an experiment to see its report. Asserts give verdicts and evaluators give quality scores.

* **Case results:** one row per test case with its verdict, filterable by **Asserts not met**, **Evaluators not met**, **Soft not met**, or **All cases**. Open a row to see the case output, the expected value, and each check's result.
* **Asserts and Evaluators:** the share of asserts met, and the average evaluator score against its threshold.
* **Configuration:** how each target node was configured for this run.

Each case gets one of these verdicts:

* **Passed:** every check that isn't soft was met.
* **Failed:** at least one check that isn't soft wasn't met.
* **Errored:** a check that isn't soft couldn't be run, or every check was skipped.

The pass rate counts passed cases against all passed, failed, and errored cases.

### Improvement suggestions

> [!NOTE]
>
> **Available starting with FlowX.AI 5.9.2**

Experiment reports show **AI-generated suggestions** for improving each target node's prompt, grouped by evaluator. Suggestions are produced when a run finishes, or finishes partially, with evaluator findings.

***

## Regression testing between agent versions

Every experiment freezes each target node's configuration at launch, so two experiments on the same test cases give you a before and after comparison of an agent change: a new prompt, a different model, or tuned retrieval.

**1. Establish the baseline**

Tag the test cases that make up your regression set, then run an experiment on that tag before changing the agent.

**2. Change the agent**

Edit the prompt, swap the model, or adjust retrieval on the AI node.

**3. Re-run the same test cases**

Start a new experiment on the same tag. The new run freezes the changed node configuration.

**4. Compare the reports**

Open both experiments from the **Experiments** tab and compare their results. A drop in asserts met or in an evaluator score is a regression; open the case results to see which inputs got worse.

> [!TIP]
>
> Grow the regression set from real runs: capture interesting AI node executions with **Add to a test case** in the run console (see [Create test cases](#create-test-cases)). Keep the organization's judge model unchanged between the two runs so the scores stay comparable.

***

## Datasets from earlier versions

Test cases replace datasets. On upgrade, existing datasets are converted:

* Each test case keeps its input and is bound to the dataset's AI node as its target.
* The dataset name becomes a tag on its test cases, so you can still select them together in an experiment.
* The expected output and retrieved context become the case output.
* Test cases captured from a node execution keep that reference.

> [!WARNING]
>
> **Converted test cases have no checks.** An experiment skips cases without an enabled check, and refuses to start if none of the selected cases has one. Open each converted case and add at least one assert or evaluator before you run it.

***

## Judge model resolution

Evaluators need a judge LLM to run their scoring prompts. The model is resolved **per organization** via an LLM capability:

* The **`EVAL`** capability is part of the org-manager LLM catalogue (display order 7, alongside the existing capabilities).
* The platform-default judge model is **`gpt-5.4-mini`**, backfilled onto every org's catalogue on upgrade.
* An admin can change the judge model per org by assigning `EVAL` to a different model in **Settings → AI Providers → Models**.

This means the same experiment can produce slightly different scores in two different orgs if they've picked different judge models. The report records which model was used, so the comparison stays like for like within one org.

> [!NOTE]
>
> The judge LLM is *not* the same as the LLM your agent uses. The agent under test can use any provider; the judge runs separately and can be a cheaper, faster model dedicated to scoring.

***

## Permissions

Evals reuse existing workflow-level permissions. There is no new role.

| Capability | Permission |
| - | - |
| List and view test cases, evaluators, experiments, and reports | `WORKFLOW_READ` |
| Create, update, or delete test cases and custom evaluators; start, cancel, or delete experiments | `WORKFLOW_EDIT` |

Permissions are enforced per workflow resource ID. A user with `WORKFLOW_EDIT` on Workflow A but not Workflow B cannot start an experiment on B's test cases.

***

## How it works

The runtime side has three pieces:

* **`integration-designer`** owns the test case, evaluator, assert, and experiment REST APIs and the experiment orchestration. It checks asserts itself, publishes judge job requests to Kafka, and writes results to MongoDB.
* **`evals-judge`** is a small Python service that consumes judge job requests, resolves the judge LLM via `organization-manager`, calls the LLM provider, and publishes scored responses.
* **`organization-manager`** owns the LLM catalogue and resolves the model bound to the `EVAL` capability for the org.

Evaluator scoring is fully async. A judge job's lifecycle:

1. `integration-designer` publishes a request on `ai.flowx.ai-platform.evals-judge.job.request.v1` with the test case, the evaluator id, and the node-config snapshot.
2. `evals-judge` consumes the request, resolves the judge model, runs the scoring prompt, and publishes the result on `ai.flowx.ai-platform.evals-judge.job.response.v1`.
3. `integration-designer` consumes the response, records the score on the case result, and pushes a progress event over SSE to any open Designer report.
4. Permanent failures (no judge model resolvable, invalid request shape, repeated timeouts) end up on `ai.flowx.ai-platform.evals-judge.job.dlq.v1` for manual operator review — the report shows the test-case row as `ERROR`.

***

## REST API surface

The Designer is the primary entry point, but the same endpoints are usable from your own tooling. All are scoped to a workspace and to the workflow resource, and require `WORKFLOW_READ` or `WORKFLOW_EDIT` on that workflow.

| Method | Endpoint | Purpose |
| - | - | - |
| GET | `/api/workflows/test-cases` | List test cases, filtered by target, tag, or name (paginated) |
| GET | `/api/workflows/test-cases/tags` | List the tags in use |
| POST | `/api/workflows/test-cases` | Create a test case |
| GET / PUT / DELETE | `/api/workflows/test-cases/{testCaseUuid}` | Fetch, replace, or delete a test case. The target can't change. |
| GET | `/api/evaluators` | List evaluators available to the workflow |
| POST | `/api/evaluators` | Create a custom evaluator |
| PUT / DELETE | `/api/evaluators/{evaluatorUuid}` | Update or delete a custom evaluator |
| GET | `/api/asserts` | List the available asserts |
| POST | `/api/workflows/experiments/start` | Start an experiment on a list of test cases |
| GET | `/api/workflows/experiments/paginated` | List experiments (paginated) |
| GET | `/api/workflows/experiments/{experimentUuid}` | Experiment report |
| GET | `/api/workflows/experiments/{experimentUuid}/cases/{resultUuid}` | One case result with all checks |
| GET | `/api/workflows/experiments/{experimentUuid}/suggestions` | Improvement suggestions |
| POST | `/api/workflows/experiments/{experimentUuid}/cancel` | Cancel a running experiment |
| DELETE | `/api/workflows/experiments/{experimentUuid}` | Delete an experiment and its results |

The `/api/workflows/datasets` endpoints from earlier versions are removed.

***

## Operator setup

Evals add a service and three Kafka topics to the AI Platform deployment.

### evals-judge service

* **Language:** Python
* **Container image:** `{{flowx-ai}}/evals-judge`
* **Dependencies:** Kafka (request / response / DLQ), `organization-manager` (LLM resolution), the LLM provider configured for the `EVAL` capability, S3-compatible storage (MinIO env vars on multimodal test cases).
* **No exposed ingress.** Communication is Kafka-only.
* **Storage:** none of its own. State lives in `integration-designer`'s MongoDB and the LLM provider.

Add `evals-judge` to your AI Platform install. See [AI Platform setup](https://docs.flowx.ai/6.x/setup-guides/ai-platform-setup.md#service-architecture) for the full Python services list.

### Kafka topics

Declare the following in your Kafka provisioning (`kafka-topics.yaml.gotmpl` for k8s, or your own topic management):

| Topic | Partitions | Purpose |
| - | - | - |
| `ai.flowx.ai-platform.evals-judge.job.request.v1` | 3 | Judge job requests from `integration-designer` |
| `ai.flowx.ai-platform.evals-judge.job.response.v1` | 3 | Scored responses back to `integration-designer` |
| `ai.flowx.ai-platform.evals-judge.job.dlq.v1` | 3 | Dead-letter queue for unscored jobs |

The `integration-designer` consumer group for the response topic is `integration-designer-evals-judge-response-group` with 3 consumer threads by default.

### EVAL capability backfill

The migration `20260513_eval_capability.xml` runs on the first `organization-manager` boot after upgrade and:

* Registers the `EVAL` capability in `llm_capability_catalog` (display order 7).
* Backfills `gpt-5.4-mini` as the model bound to `EVAL` in `llm_model_catalog` (platform default).
* Backfills the same binding into every existing organization's `llm_model` table so evals work out-of-the-box on upgrade without manual configuration.

If your org uses a non-OpenAI provider, change the bound model in **Settings → AI Providers → Models** after the upgrade.

***

## Limitations

* **Workflow-scoped only.** Test cases and custom evaluators belong to a single workflow. Sharing them across workflows isn't supported.
* **AI nodes only.** A test case targets one AI node. Document Generation, Context Retrieval, and Deep Research nodes don't support test cases.
* **Judge model is org-wide.** Picking a different judge per evaluator or per experiment is not supported.
* **No similarity assert.** Asserts check exact values and JSON equality. Levenshtein similarity from earlier versions has no replacement.
* **DLQ is manual.** There is no automatic retry for jobs landing on the DLQ; an operator must inspect and decide whether to retry.

***

## Related resources

> [card] [AI Platform setup](https://docs.flowx.ai/6.x/setup-guides/ai-platform-setup.md)
>
> Install evals-judge alongside the rest of the AI Platform.

> [card] [AI providers](https://docs.flowx.ai/6.x/ai-platform/ai-providers.md)
>
> Configure the LLM bound to the `EVAL` capability for your org.

> [card] [Agent Builder overview](https://docs.flowx.ai/6.x/ai-platform/agent-builder/overview.md)
>
> Build the AI agents whose outputs you'll test.

> [card] [Chat-driven workflows](https://docs.flowx.ai/6.x/ai-platform/chat-driven-workflows.md)
>
> Chat-driven workflow nodes you can write test cases for.


## Related topics

- [FlowX.AI 5.9.0 Release Notes](https://docs.flowx.ai/release-notes/v5.x/v5.9.0-june-2026/v5.9.0-june-2026.md)
- [FlowX.AI 5.10.0 Release Notes](https://docs.flowx.ai/release-notes/v5.x/v5.10.0-july-2026/v5.10.0-july-2026.md)
- [FlowX.AI 6.0.0 Release Notes](https://docs.flowx.ai/release-notes/v6.x/v6.0.0-october-2026/v6.0.0-october-2026.md)
- [Speech to Text setup](https://docs.flowx.ai/6.x/setup-guides/speech-to-text-setup.md)
- [Error glossary](https://docs.flowx.ai/6.x/docs/resources/error-glossary.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.