Evaluations

​

Overview

An evaluation (eval) scores the output of an AI node against a known expected outcome. Use evals to:

• Tune prompts and retrieval — change a system prompt or KB chunk size, rerun the same experiment, and see whether your scores improved.
• Regression-guard agents — before promoting a new agent version, run the dataset of “hard” past inputs and compare scores against the previous version.
• Pick a model — run the same dataset against two different LLMs and compare correctness, conciseness, and cost.

Evals are workflow-centric. Each dataset is attached to a specific AI node in a specific workflow; experiments launch that workflow against the dataset and score the node’s outputs.

---

​

Core concepts

---

​

Get started

---

​

Create a dataset

A dataset belongs to a single AI node on a single workflow. Two datasets on the same node must have unique names. Fields: The dataset persists a schema snapshot of the node’s input/output at the time of last edit. This is what’s used when a workflow is changed underneath an existing dataset — the experiment runs against the snapshot, not the live node, so older runs remain reproducible.

---

​

Add test cases

A test case has three core fields:

• Input — the data the AI node receives.
• Expected output — what a correct response looks like. Required by exact-match and JSON-match evaluators; optional but recommended for LLM-as-judge.
• Context — optional retrieved context (KB chunks, retrieval results) used to score RAG evaluators.

Two ways to add them:

---

​

Launch an experiment

An experiment binds together:

• One or more datasets (all must be on the same AI node)
• One or more evaluators
• A node configuration snapshot (frozen at launch time)

When you click Start, the workflow runs end-to-end against every test case. Each output produces one judge job per (test case × evaluator), published to the evals-judge.job.request Kafka topic. evals-judge resolves the configured judge LLM, runs the scoring prompt, and returns a score + reasoning on evals-judge.job.response. Experiment status moves through: The Designer streams progress updates as scoring jobs complete, so the report fills in live without a full reload.

​

Multiple datasets

An experiment can run against multiple datasets at once (datasets sharing the same AI node). Overall metrics are aggregated across all of them, and the report adds a dataset column so you can see how each case scored per dataset.

---

​

View results

Each experiment produces a report with two views:

​

Aggregate scores

A row per evaluator with the mean (or pass-rate) across all test cases. Useful for “is this prompt better than the last one?” comparisons.

​

Per-test-case details

A row per test case showing:

• The input, the model’s actual output, and the expected output side-by-side.
• One score column per evaluator selected for the experiment.
• For LLM-as-judge evaluators, the judge’s reasoning text (the model’s own explanation of the score).
• Pass/fail badges against each evaluator’s threshold.

Click into a row to see the full prompt, response, and any retrieved context — this is the screen you use to debug “why is this case failing?”

​

Improvement suggestions

Experiment reports show AI-generated suggestions for improving the node’s prompt, grouped per LLM-as-judge evaluator, beneath the score cards — turning a low score into a concrete next edit.

---

​

Evaluator catalogue

The catalogue includes 10 built-in evaluators in three categories. It is read-only — user-defined evaluators are planned for a later release.

​

General quality

A judge LLM reads the model output (and optional expected output) and scores it on a single rubric. Score is numeric 0.0–1.0. In the Designer these appear under the General Quality group.

​

RAG

Evaluators for RAG workflows, covering both the retrieval and the answer. Score is numeric 0.0–1.0.

​

Code / deterministic

Deterministic, no LLM call. Useful for structured outputs.

​

Score types and thresholds

• Numeric evaluators return a score in [0.0, 1.0] and ship with a defaultThreshold (e.g. 0.7). The report flags a row as failing if the score is below the threshold.
• Binary evaluators return pass or fail. No threshold.

---

​

Judge model resolution

LLM-as-judge and RAG evaluators need a judge LLM to run their scoring prompts. The model is resolved per organization via a new LLM capability:

• The EVAL capability is part of the org-manager LLM catalogue (display order 7, alongside the existing capabilities).
• The platform-default judge model is gpt-5.4-mini, backfilled onto every org’s catalogue on upgrade.
• An admin can change the judge model per org by assigning EVAL to a different model in Settings → AI Providers → Models.

This means the same experiment can produce slightly different scores in two different orgs if they’ve picked different judge models — the report records which model was used, so the comparison stays apples-to-apples within one org.

---

​

Permissions

Evals reuse existing workflow-level permissions — there is no new role. Permissions are enforced per workflow resource ID — a user with WORKFLOW_EDIT on Workflow A but not Workflow B cannot launch an experiment against B’s nodes.

---

​

How it works

The runtime side has three pieces:

• integration-designer owns the dataset, test-case, and experiment REST APIs and the experiment orchestration. It publishes judge job requests to Kafka and writes results to MongoDB.
• evals-judge is a small Python service that consumes judge job requests, resolves the judge LLM via organization-manager, calls the LLM provider, and publishes scored responses.
• organization-manager owns the LLM catalogue and resolves the model bound to the EVAL capability for the org.

Scoring is fully async. A judge job’s lifecycle:

1. integration-designer publishes a request on ai.flowx.ai-platform.evals-judge.job.request.v1 with the test case, the evaluator id, and the node-config snapshot.
2. evals-judge consumes the request, resolves the judge model, runs the scoring prompt, and publishes the result on ai.flowx.ai-platform.evals-judge.job.response.v1.
3. integration-designer consumes the response, persists the score on the test-case row, and pushes a progress event over SSE to any open Designer report.
4. Permanent failures (no judge model resolvable, invalid request shape, repeated timeouts) end up on ai.flowx.ai-platform.evals-judge.job.dlq.v1 for manual operator review — the report shows the test-case row as ERROR.

---

​

REST API surface

The Designer is the primary entry point, but the same endpoints are usable from your own tooling. All are scoped to a workspace and require WORKFLOW_READ / WORKFLOW_EDIT on the relevant workflow.

---

​

Operator setup

Evals add a new service and three new Kafka topics to the AI Platform deployment.

​

evals-judge service

• Language: Python
• Container image: {{flowx-ai}}/evals-judge
• Dependencies: Kafka (request / response / DLQ), organization-manager (LLM resolution), the LLM provider configured for the EVAL capability, S3-compatible storage (MinIO env vars on multimodal test cases).
• No exposed ingress. Communication is Kafka-only.
• Storage: none of its own — state lives in integration-designer’s MongoDB and the LLM provider.

Add evals-judge to your AI Platform install. See AI Platform setup for the full Python services list.

​

Kafka topics

Declare the following in your Kafka provisioning (kafka-topics.yaml.gotmpl for k8s, or your own topic management): The integration-designer consumer group for the response topic is integration-designer-evals-judge-response-group with 3 consumer threads by default.

​

EVAL capability backfill

The migration 20260513_eval_capability.xml runs on the first organization-manager boot after upgrade and:

• Registers the EVAL capability in llm_capability_catalog (display order 7).
• Backfills gpt-5.4-mini as the model bound to EVAL in llm_model_catalog (platform default).
• Backfills the same binding into every existing organization’s llm_model table so evals work out-of-the-box on upgrade without manual configuration.

If your org uses a non-OpenAI provider, change the bound model in Settings → AI Providers → Models after the upgrade.

---

​

Limitations

• No user-defined evaluators. The 10-evaluator catalogue is read-only. Custom rubrics and custom scoring prompts are planned for a later release.
• Workflow-scoped only. Datasets bind to a single AI node on a single workflow. Cross-workflow shared datasets are out of scope.
• Judge model is org-wide. Picking a different judge per evaluator or per experiment is not supported.
• No retroactive backfill across workflow rewrites. If you change the AI node’s contract significantly, existing datasets keep their snapshot — they don’t fail, but they may no longer reflect the current workflow’s behaviour. Create a new dataset.
• DLQ is manual. There is no automatic retry for jobs landing on the DLQ; an operator must inspect and decide whether to retry.

---

​

Related resources