> ## Documentation Index
> Fetch the complete documentation index at: https://docs.flowx.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Kafka migration
> One mandatory new header, new Kafka topics that did not exist in 5.1.x, a topic-ownership move, and the custom-producer audit you must run before upgrading.
5.1.x deployments interact with FlowX over Kafka in two ways: messages produced **by** FlowX services (process start, task events, audit, document operations) and messages produced **into** FlowX by your own integrations. The 5.1 → 5.9 jump introduces one breaking change that affects custom producers, several new topics introduced by new services, and one topic-ownership move that affects Kafka ACLs.
***
## Mandatory `Fx-Organization-Id` header
On 5.9.x, every FlowX Kafka consumer validates the `Fx-Organization-Id` header on every incoming message. Messages without this header are rejected immediately and not retried.
This is the single most likely cause of silent breakage post-upgrade for customers who maintain custom Kafka producers (process start, task operations, engine responses, integration callbacks).
### What to set
Set the header value to the Organization UUID provided by FlowX during organization registration. The same value you configure as the `ORGANIZATION_ID` environment variable on every FlowX service. See [Prerequisites](./prerequisites#get-your-organization-uuid).
```java theme={"system"}
producerRecord.headers().add("Fx-Organization-Id",
"26ef7b5f-b463-4375-88c8-xxxxxxxxxxxx".getBytes(StandardCharsets.UTF_8));
```
### Where this applies
| Surface | Behavior in 5.9.0 |
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Kafka topics consumed by FlowX | Header is **mandatory**. Missing → message rejected, not retried. |
| Kafka responses sent **back** to FlowX | Preserve all received headers (including `Fx-Organization-Id`) when responding from your connector. |
| REST API endpoints | Header is read into security context when present, but **not enforced** in 5.9.0. Enforcement on authenticated REST requests is tracked separately and will land in a later release. |
Pre-existing 5.1.x integrations that simply forward Kafka messages back to FlowX without copying the headers will break. Audit every custom Kafka producer and connector before the upgrade window.
### Affected consumer surfaces
Validation runs on **every** FlowX Kafka listener. Concretely, custom producers that target the following topic families must add the header:
* Process start (`*.core.trigger.start.process.*`, `*.core.trigger.start-for-event.process.*`)
* Process operations (`*.core.trigger.operation.*`, `*.core.trigger.operations.bulk.*`): task assign/unassign, hold/unhold, terminate, including bulk
* Connector responses (`*.engine.receive.*`)
* Webhook ingestion, document plugin triggers, integration sync topics
***
## New Kafka topics that did not exist in 5.1.x
The 5.1 → 5.9 jump adds new microservices (`organization-manager`, `webhook-gateway`, `license`) and new features (Knowledge Base, Multi-file Upload, UI Flow session variables, Email Trigger). Each brings its own topics. For the complete list with producer/consumer mapping, see the deployment guidelines linked at the bottom of this page.
The high-level groups your platform/SRE team should be aware of:
| Theme | Notable topics |
| ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `organization-manager` + Document plugin enhancements | `ai.flowx.organization.events.v1`, `ai.flowx.application-version.resources-usages.*`, `ai.flowx.plugin.document.trigger.persist.document.*`, `ai.flowx.build.runtime-data.v1` |
| `webhook-gateway` + UI Flow session variables | `ai.flowx.core.trigger.start-for-event.process.v1`, `ai.flowx.core.trigger.ui-flow.update.v1`, `ai.flowx.plugin.document.trigger.persist.document.session.v1` |
| AI Providers UI (LLM config) | `ai.flowx.llm.config.changed.v1` |
| `license` service + KB internal Kafka path + bulk document persistence | `ai.flowx.license.events.v1`, `ai.flowx.ai-platform.knowledgebase.internal.embedding-request.v1` / `-response.v1`, `ai.flowx.plugin.document.trigger.persist.document.session.bulk.v1` |
If you manage Kafka topic creation manually (rather than letting FlowX auto-create), pre-create the topics listed in each release's deployment guidelines before deploying the corresponding service versions.
***
## New producer on `WorkspaceCreatedEvent`: `organization-manager`
`organization-manager` now publishes `WorkspaceCreatedEvent` as part of the organization-provisioning flow. `authorization-system` continues to publish the event on regular workspace creation, so on 5.9.x the topic has **two producers**, not one.
If you manage Kafka ACLs at the topic level, grant producer access to **both** services:
* `authorization-system`. Publishes on user-driven workspace create (unchanged from 5.1.x).
* `organization-manager`. Publishes during org provisioning (new in 5.9.x).
Consumers of the topic do not change. If you rely on FlowX's default Kafka ACL provisioning (no manual ACL management), no action is required.
***
## Knowledge Base indexer ↔ embedder moves to Kafka
On 5.9.x, the Knowledge Base indexer → embedder path uses a Kafka request/response pair instead of synchronous gRPC:
* `ai.flowx.ai-platform.knowledgebase.internal.embedding-request.v1` (indexer producer, embedder consumer)
* `ai.flowx.ai-platform.knowledgebase.internal.embedding-response.v1` (embedder producer, indexer consumer)
This is internal to FlowX. Your integrations are unaffected. The implication for operators is that the embedder no longer needs to receive gRPC calls from the indexer; both services connect to Kafka for KB indexing traffic.
***
## Custom-producer audit checklist
Before the upgrade window, walk through every Kafka integration that produces into FlowX:
List every system that publishes to a FlowX Kafka topic. Common sources: backends that start processes by name, task-management automations, connector implementations that respond to engine requests, scheduled jobs that emit events.
For each producer, add the header before sending. The value is your Organization UUID (same as the `ORGANIZATION_ID` env var).
Connectors that respond to FlowX engine requests must copy **all** received Kafka headers (including `Fx-Organization-Id`, `Fx-Workspace-Id`, `fxContext`, `Fx-AppId`) into the response. The simplest pattern: `originalHeaders.forEach(producerRecord.headers()::add)`.
Run end-to-end with a 5.9.x staging environment before production. The failure mode is non-retryable rejection: messages are dropped, not queued for retry, so the breakage stays silent unless you read the consumer logs.
If you manage ACLs manually, switch the `WorkspaceCreatedEvent` producer ACL from `authorization-system` to `organization-manager`. Pre-create any new topics from the table above that aren't already provisioned.
***
## Related resources
Full reference for required Kafka headers on the 5.9.x baseline, including the `Fx-Organization-Id`, `Fx-Workspace-Id`, and `jwt` headers.
Where to find the Organization UUID, workspace ID, project ID, and other identifiers required in Kafka and REST integrations.
Component versions, Kafka topic ownership, and env vars for the 5.9.x baseline.
Get your Organization UUID and complete the pre-upgrade inventory before working through this page.