Evaluations - FlowX.AI

Write test cases for an AI node in a workflow, check its output with deterministic asserts, score its quality with LLM evaluators, and run an experiment over a tagged set of cases. Judge scoring runs asynchronously in the evals-judge service.

Overview

An evaluation (eval) checks how an AI node behaves on a known input. Use evals to:

Tune prompts and retrieval. Change a system prompt or a knowledge base setting, rerun the same test cases, and see whether the results improved.
Guard against regressions. Before you commit a new agent version, run your set of hard past inputs and compare the results with the previous run.
Pick a model. Run the same test cases against two LLMs and compare the verdicts and quality scores.

Evals are workflow-centric. Open a workflow and select the flask icon (Test Cases) in the left sidebar. The page has three tabs: Evaluators, Cases, and Experiments.

The Test Cases page of a workflow with the Evaluators tab open, listing the General Quality evaluators Answer Relevance, Conciseness, Correctness, and Hallucination, each with Numeric scoring and a 70% pass threshold

Core concepts

Concept	What it is
Test case	A JSON input for one AI node (the case’s target), plus the checks its output must pass. Grouped with tags.
Assert	A deterministic check on the node output, such as equals an expected value. Gives a pass or fail verdict.
Evaluator	An LLM judge that scores the quality of the output, such as correctness or groundedness, against a pass threshold.
Experiment	A run of a selected set of test cases. Freezes the configuration of each target node and records a result for every case.

A test case needs at least one check: an assert, an evaluator, or both.

Get started

Open the Test Cases page

In a workflow that contains an AI node, select the flask icon (Test Cases) in the left sidebar. You can also select the flask button on an AI node on the canvas, which opens the Cases tab filtered to that node.

Create a test case

On the Cases tab, click New test case. Enter a Case name, pick the Target node, and add Tags to group the case with others.

Enter the input and add checks

Enter the JSON Input. Under Test case, use Add to add an Assert, an Eval, or both.

Run the case and save it

Click Run to run the node on its own over the input. Saving keeps the output as the case output. Click Create case.

Run an experiment

On the Experiments tab, click New experiment, name the run, select the tags or test cases to include, and click Run tests.

Read the results

Open the experiment to see assert verdicts and evaluator scores per target, with a result for every case.

Create test cases

A test case belongs to one workflow and tests one AI node. Every AI node type supports test cases except Document Generation, Context Retrieval, and Deep Research.

Field	Required	Description
Case name	Yes	Identifies the case in lists and reports. Up to 200 characters.
Target	Yes	The AI node the case tests. Can’t be changed after the case is created.
Tags	No	Labels that group cases. Experiments select test cases by tag. Type a new tag to create it.
Input	Yes	The JSON data the node runs on. Must be valid JSON. `null`, `false`, `0`, and empty collections are valid values.

You can add test cases in three ways:

Manually
From a workflow run
From the canvas

On the Cases tab, click New test case and fill in the fields. Under Test case, click Run to run the node on its own over the input, then add asserts and evaluators that are checked against that output.If you save without running, the case has no output yet. The asserts and evaluators are checked when the case runs in an experiment.

The New test case dialog with Case name, Tags, and a Target set to an AI node, showing one equals assert that checks the whole node output against an expected value

Use the Cases tab filters (All Tags, All Targets, and Search by name) to find cases. Each row shows the case target, its number of asserts and evaluators, and the result of its last run. Test cases and custom evaluators travel with the workflow when you clone, export, import, or merge it. A clone or import doesn’t add test cases to a workflow that already has some.

Add asserts

An assert checks the node output against a value you expect and gives a verdict: met or not met. Asserts don’t call an LLM.

Assert	What it checks
equals	The output, or a value inside it, equals the expected value: a number, `true` or `false`, or text.
equals as JSON	The output equals the expected JSON. The number type matters, so `1` and `1.0` are different.

Mark an assert Soft to report it without failing the case. A soft assert that isn’t met shows in the report under Soft not met.

The Code evaluators are now asserts. The Exact match and JSON match evaluators from earlier versions are replaced by the equals and equals as JSON asserts. Levenshtein similarity has no replacement.

Add evaluators

An evaluator is an LLM judge that scores the output from 0 to 1 against a pass threshold. In a test case, add one with Add → Eval, pick the evaluator, and optionally change the pass threshold for that case. The Evaluators tab lists the evaluators available in the workflow: the built-in ones, grouped by category, and any you create. The built-in evaluators can’t be edited or deleted.

General quality

A judge LLM reads the node output, and the expected output when there is one, and scores it on a single rubric. All four use Numeric scoring with a default pass threshold of 70%.

Evaluator	What it scores
Correctness	Whether the output contains the same key facts as the expected output.
Conciseness	Whether the output is appropriately brief. Penalizes padding and repetition.
Hallucination	Whether the output makes claims not supported by the provided context or the expected output.
Answer Relevance	Whether the output answers what the input asked, regardless of factual correctness.

RAG

Evaluators for retrieval-augmented generation, covering both retrieval and the answer. They apply to a Custom Agent node that uses a knowledge base. All three use Numeric scoring with a default pass threshold of 70%.

Evaluator	What it scores
RAG Groundedness	Whether every claim in the output traces back to the retrieved context, rather than the model’s prior knowledge.
RAG Helpfulness	How useful the answer is to the end user, considering tone, completeness, and actionability.
RAG Retrieval Relevance	How relevant the retrieved documents are to the question. A quality check for the retrieval step.

RAG Groundedness and RAG Retrieval Relevance need retrieved context. Without it, the check errors instead of scoring.

Custom

Write your own evaluator when none of the built-in rubrics measures what you care about. A custom evaluator is scored by the judge LLM following instructions you supply, and appears under the Custom group on the Evaluators tab, after the built-in ones. Custom evaluators belong to one workflow. Click New evaluator on the Evaluators tab to create one.

Field	Required	What it does
Start from a template	No	Prefills the form from an existing evaluator, instructions included, so you can adapt a built-in rubric instead of starting from a blank prompt.
Name	Yes	Identifies the evaluator on the Evaluators tab and in experiment reports. Must be unique in the workflow.
Description	No	Records what the evaluator measures, for whoever reads the list next.
Instructions	Yes	The prompt sent to the judge model for every test case output. Tell the judge what to look for and how to score it.
Scoring	Yes	Numeric for a score from 0 to 1, or Binary for pass or fail. Numeric is the default.
Pass threshold	Only when scoring is Numeric	The score a case must reach to pass. Must be between `0` and `1`.

Binary scoring has no threshold. Switch Scoring to Binary and the Pass threshold field is turned off, since pass or fail is already the result.

Unlike the built-in evaluators, the ones you create can be edited and deleted. A deleted evaluator is no longer available for new experiments. Past experiment reports keep their results.

Run an experiment

An experiment runs a selected set of test cases. Each case runs against its own target node, so one experiment can cover several AI nodes in the workflow. The report groups the results by target.

Start a new experiment

On the Experiments tab, click New experiment and enter a Name for the run.

Select the test cases

Select tags to include the test cases that carry them, then use Edit selection to add or remove individual cases. Cases whose target node was deleted are skipped.

Run the tests

Click Run tests. The configuration of each target node is frozen for this run, and the report fills in live as the checks complete.

The New experiment dialog with a Name field, a Test cases section to select tags and edit the selection, and a Run tests button

An experiment doesn’t start if no selected case has an enabled check, if every selected case targets a node that no longer exists, or if a case input isn’t valid JSON. Experiment status moves through:

Status	Meaning
`STARTED`	The experiment is running.
`FINISHED`	Every case ran without errors. Individual cases can still have failed their checks.
`PARTIAL`	Some cases errored. The results of the other cases are available.
`FAILED`	Every case errored.
`CANCELLED`	Cancelled by a user while running. Cases still in progress are abandoned; results already recorded are kept.

You can cancel an experiment only while it is running. Deleting an experiment removes its results and keeps the test cases.

View results

Open an experiment to see its report. Asserts give verdicts and evaluators give quality scores.

Case results: one row per test case with its verdict, filterable by Asserts not met, Evaluators not met, Soft not met, or All cases. Open a row to see the case output, the expected value, and each check’s result.
Asserts and Evaluators: the share of asserts met, and the average evaluator score against its threshold.
Configuration: how each target node was configured for this run.

Each case gets one of these verdicts:

Passed: every check that isn’t soft was met.
Failed: at least one check that isn’t soft wasn’t met.
Errored: a check that isn’t soft couldn’t be run, or every check was skipped.

The pass rate counts passed cases against all passed, failed, and errored cases.

Improvement suggestions

Available starting with FlowX.AI 5.9.2

Experiment reports show AI-generated suggestions for improving each target node’s prompt, grouped by evaluator. Suggestions are produced when a run finishes, or finishes partially, with evaluator findings.

Regression testing between agent versions

Every experiment freezes each target node’s configuration at launch, so two experiments on the same test cases give you a before and after comparison of an agent change: a new prompt, a different model, or tuned retrieval.

Establish the baseline

Tag the test cases that make up your regression set, then run an experiment on that tag before changing the agent.

Change the agent

Edit the prompt, swap the model, or adjust retrieval on the AI node.

Re-run the same test cases

Start a new experiment on the same tag. The new run freezes the changed node configuration.

Compare the reports

Open both experiments from the Experiments tab and compare their results. A drop in asserts met or in an evaluator score is a regression; open the case results to see which inputs got worse.

Grow the regression set from real runs: capture interesting AI node executions with Add to a test case in the run console (see Create test cases). Keep the organization’s judge model unchanged between the two runs so the scores stay comparable.

Datasets from earlier versions

Test cases replace datasets. On upgrade, existing datasets are converted:

Each test case keeps its input and is bound to the dataset’s AI node as its target.
The dataset name becomes a tag on its test cases, so you can still select them together in an experiment.
The expected output and retrieved context become the case output.
Test cases captured from a node execution keep that reference.

Converted test cases have no checks. An experiment skips cases without an enabled check, and refuses to start if none of the selected cases has one. Open each converted case and add at least one assert or evaluator before you run it.

Judge model resolution

Evaluators need a judge LLM to run their scoring prompts. The model is resolved per organization via an LLM capability:

The EVAL capability is part of the org-manager LLM catalogue (display order 7, alongside the existing capabilities).
The platform-default judge model is gpt-5.4-mini, backfilled onto every org’s catalogue on upgrade.
An admin can change the judge model per org by assigning EVAL to a different model in Settings → AI Providers → Models.

This means the same experiment can produce slightly different scores in two different orgs if they’ve picked different judge models. The report records which model was used, so the comparison stays like for like within one org.

The judge LLM is not the same as the LLM your agent uses. The agent under test can use any provider; the judge runs separately and can be a cheaper, faster model dedicated to scoring.

Permissions

Evals reuse existing workflow-level permissions. There is no new role.

Capability	Permission
List and view test cases, evaluators, experiments, and reports	`WORKFLOW_READ`
Create, update, or delete test cases and custom evaluators; start, cancel, or delete experiments	`WORKFLOW_EDIT`

Permissions are enforced per workflow resource ID. A user with WORKFLOW_EDIT on Workflow A but not Workflow B cannot start an experiment on B’s test cases.

How it works

The runtime side has three pieces:

integration-designer owns the test case, evaluator, assert, and experiment REST APIs and the experiment orchestration. It checks asserts itself, publishes judge job requests to Kafka, and writes results to MongoDB.
evals-judge is a small Python service that consumes judge job requests, resolves the judge LLM via organization-manager, calls the LLM provider, and publishes scored responses.
organization-manager owns the LLM catalogue and resolves the model bound to the EVAL capability for the org.

Evaluator scoring is fully async. A judge job’s lifecycle:

integration-designer publishes a request on ai.flowx.ai-platform.evals-judge.job.request.v1 with the test case, the evaluator id, and the node-config snapshot.
evals-judge consumes the request, resolves the judge model, runs the scoring prompt, and publishes the result on ai.flowx.ai-platform.evals-judge.job.response.v1.
integration-designer consumes the response, records the score on the case result, and pushes a progress event over SSE to any open Designer report.
Permanent failures (no judge model resolvable, invalid request shape, repeated timeouts) end up on ai.flowx.ai-platform.evals-judge.job.dlq.v1 for manual operator review — the report shows the test-case row as ERROR.

REST API surface

The Designer is the primary entry point, but the same endpoints are usable from your own tooling. All are scoped to a workspace and to the workflow resource, and require WORKFLOW_READ or WORKFLOW_EDIT on that workflow.

Method	Endpoint	Purpose
GET	`/api/workflows/test-cases`	List test cases, filtered by target, tag, or name (paginated)
GET	`/api/workflows/test-cases/tags`	List the tags in use
POST	`/api/workflows/test-cases`	Create a test case
GET / PUT / DELETE	`/api/workflows/test-cases/{testCaseUuid}`	Fetch, replace, or delete a test case. The target can’t change.
GET	`/api/evaluators`	List evaluators available to the workflow
POST	`/api/evaluators`	Create a custom evaluator
PUT / DELETE	`/api/evaluators/{evaluatorUuid}`	Update or delete a custom evaluator
GET	`/api/asserts`	List the available asserts
POST	`/api/workflows/experiments/start`	Start an experiment on a list of test cases
GET	`/api/workflows/experiments/paginated`	List experiments (paginated)
GET	`/api/workflows/experiments/{experimentUuid}`	Experiment report
GET	`/api/workflows/experiments/{experimentUuid}/cases/{resultUuid}`	One case result with all checks
GET	`/api/workflows/experiments/{experimentUuid}/suggestions`	Improvement suggestions
POST	`/api/workflows/experiments/{experimentUuid}/cancel`	Cancel a running experiment
DELETE	`/api/workflows/experiments/{experimentUuid}`	Delete an experiment and its results

The /api/workflows/datasets endpoints from earlier versions are removed.

Operator setup

Evals add a service and three Kafka topics to the AI Platform deployment.

evals-judge service

Language: Python
Container image: {{flowx-ai}}/evals-judge
Dependencies: Kafka (request / response / DLQ), organization-manager (LLM resolution), the LLM provider configured for the EVAL capability, S3-compatible storage (MinIO env vars on multimodal test cases).
No exposed ingress. Communication is Kafka-only.
Storage: none of its own. State lives in integration-designer’s MongoDB and the LLM provider.

Add evals-judge to your AI Platform install. See AI Platform setup for the full Python services list.

Kafka topics

Declare the following in your Kafka provisioning (kafka-topics.yaml.gotmpl for k8s, or your own topic management):

Topic	Partitions	Purpose
`ai.flowx.ai-platform.evals-judge.job.request.v1`	3	Judge job requests from `integration-designer`
`ai.flowx.ai-platform.evals-judge.job.response.v1`	3	Scored responses back to `integration-designer`
`ai.flowx.ai-platform.evals-judge.job.dlq.v1`	3	Dead-letter queue for unscored jobs

The integration-designer consumer group for the response topic is integration-designer-evals-judge-response-group with 3 consumer threads by default.

EVAL capability backfill

The migration 20260513_eval_capability.xml runs on the first organization-manager boot after upgrade and:

Registers the EVAL capability in llm_capability_catalog (display order 7).
Backfills gpt-5.4-mini as the model bound to EVAL in llm_model_catalog (platform default).
Backfills the same binding into every existing organization’s llm_model table so evals work out-of-the-box on upgrade without manual configuration.

If your org uses a non-OpenAI provider, change the bound model in Settings → AI Providers → Models after the upgrade.

Limitations

Workflow-scoped only. Test cases and custom evaluators belong to a single workflow. Sharing them across workflows isn’t supported.
AI nodes only. A test case targets one AI node. Document Generation, Context Retrieval, and Deep Research nodes don’t support test cases.
Judge model is org-wide. Picking a different judge per evaluator or per experiment is not supported.
No similarity assert. Asserts check exact values and JSON equality. Levenshtein similarity from earlier versions has no replacement.
DLQ is manual. There is no automatic retry for jobs landing on the DLQ; an operator must inspect and decide whether to retry.

AI Platform setup

Install evals-judge alongside the rest of the AI Platform.

AI providers

Configure the LLM bound to the EVAL capability for your org.

Agent Builder overview

Build the AI agents whose outputs you’ll test.

Chat-driven workflows

Chat-driven workflow nodes you can write test cases for.

​Overview

​Core concepts

​Get started

​Create test cases

​Add asserts

​Add evaluators

​General quality

​RAG

​Custom

​Run an experiment

​View results

​Improvement suggestions

​Regression testing between agent versions

​Datasets from earlier versions

​Judge model resolution

​Permissions

​How it works

​REST API surface

​Operator setup

​evals-judge service

​Kafka topics

​EVAL capability backfill

​Limitations

​Related resources

AI Platform setup

AI providers

Agent Builder overview

Chat-driven workflows

Overview

Core concepts

Get started

Create test cases

Add asserts

Add evaluators

General quality

RAG

Custom

Run an experiment

View results

Improvement suggestions

Regression testing between agent versions

Datasets from earlier versions

Judge model resolution

Permissions

How it works

REST API surface

Operator setup

evals-judge service

Kafka topics

EVAL capability backfill

Limitations

Related resources