Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.flowx.ai/llms.txt

Use this file to discover all available pages before exploring further.

Policies are the rules your runtime has to obey. Observatory’s policy engine defines those rules, assigns them to apps, evaluates them against actual runs, and rolls the results up into a compliance score per app and per policy.

​
Concepts

ModelWhat it represents
PolicyA single rule. Carries severity (low, medium, high, critical) and an evaluator expression.
PolicyPackA named bundle of policies that ship together (e.g. “PII basics”, “EU AI Act high-risk app”).
PolicyAssignmentA pack assigned to one app.
PolicyEvaluationOne evaluation of one policy against one run. Stores the outcome and any captured violation context.

​
Severities

SeverityUse forDefault behaviour
criticalThings that must never happen — PII leak, credential exposurePage on first violation.
highSerious deviations from expected behaviourAlert, surfaces on Risk Dashboard.
mediumQuality issues — refusals, low confidenceCounted, no immediate alert.
lowStyle or formattingCounted only.

​
Creating a policy

1

Open Policies → Policies

Click Add policy.
2

Name and describe

The description is what reviewers see during an audit. Be specific.
3

Pick a severity

Critical and high contribute to the Compliance roll-up score; medium and low are reported but don’t block.
4

Define the evaluator

Built-in evaluators cover PII detection, toxicity, prompt-injection patterns, and a few model-specific checks. Custom evaluators are user-defined Python predicates.
5

Test against past runs

Use Evaluate to dry-run the policy against the last N runs of a chosen app. The preview shows which runs would have violated.

​
Bundling into packs

Policies travel in packs. A pack is what you assign — never an individual policy. Typical packs:
  • PII basics — emails, phone numbers, government IDs.
  • EU AI Act high-risk — full set of obligations for high-risk apps.
  • Internal data hygiene — no production data in development environments.
  • Customer-facing assistant — refusal patterns, escalation triggers.
Create a pack from Policies → Packs, add policies to it, then assign the pack to one or more apps under Assignments.

​
Compliance roll-up

The Compliance tab scores each app against the policies assigned to it. The score weights by severity — critical violations dominate, low violations contribute proportionally. 
GET /api/policies/compliance?app_id=...
Returns:
  • score — 0–100
  • violations_by_severity
  • top_violated_policies — useful for prioritising remediation

​
API

EndpointUse
GET /api/policiesList policies.
POST /api/policiesCreate.
GET /api/policies/packsList packs.
POST /api/policies/packsCreate.
POST /api/policies/assignmentsAssign a pack to an app.
POST /api/policies/evaluateRun evaluation against a run or batch.
GET /api/policies/complianceRoll-up score.
FastAPI route ordering matters. Static routes (/evaluate, /compliance) must be defined before the /{org_id} catch-all. If you fork the router, keep that order.

Evidence

Capture the artefacts that prove policy enforcement.

Risk Dashboard

See how policy compliance feeds the per-app risk score.
Last modified on June 2, 2026