> ## Documentation Index
> Fetch the complete documentation index at: https://docs.flowx.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# FlowX.AI data architecture: which service uses which store

> See which data stores FlowX.AI uses, what each one holds, and which services read and write each PostgreSQL database, MongoDB database, and bucket.

## Overview

FlowX.AI uses multiple specialized data stores, each chosen for specific workload characteristics: PostgreSQL for process definitions and instance state, MongoDB for runtime configuration and the FlowX Database, Redis for caching and sessions, Kafka for event streaming, S3-compatible object storage for files and documents, Elasticsearch for audit logs and search indexes, and Qdrant for vector embeddings and semantic retrieval. They fall into two categories:

* **Embedded data components**: shipped with the platform, managed by FlowX.AI
* **Platform data services**: third-party dependencies deployed by your team

```mermaid theme={"dark"}
flowchart LR
    FX["FlowX.AI Platform"]

    FX -->|"process definitions, instance state,\nplatform config, metadata"| PG[("PostgreSQL")]
    FX -->|"runtime config, workflow state,\nFlowX Database"| MONGO[("MongoDB")]
    FX -->|"cache, sessions,\ndistributed locks"| REDIS[("Redis")]
    FX -->|"event streaming,\nasync messaging"| KAFKA[("Kafka")]
    FX -->|"files, documents,\nbinary assets"| S3[("S3-compatible\nobject storage")]
    FX -->|"audit logs,\nsearch indexes"| ES[("Elasticsearch")]
    FX -->|"vector embeddings,\nsemantic retrieval"| QD[("Qdrant")]

    style PG fill:#336791,color:#fff
    style MONGO fill:#4DB33D,color:#000
    style REDIS fill:#DC382D,color:#fff
    style KAFKA fill:#231F20,color:#fff
    style S3 fill:#FF9900,color:#000
    style ES fill:#FED10A,color:#000
    style QD fill:#DC244C,color:#fff
```

***

## Embedded data components

> [!NOTE]
>
> Embedded data components are delivered as part of the FlowX.AI platform (Docker images + Helm charts). They run on Kubernetes only, and their versioning is managed by FlowX.AI. These components are included in the standard deployment and cannot be replaced by alternative implementations.

> [card] [Qdrant](https://qdrant.tech/documentation/)
>
> **Type:** Vector database
>
> Qdrant is used by the AI Platform RAG path (`knowledgebase-rag`, `embedder`, `knowledgebase-indexer-v2`) and by AI agents for semantic retrieval. It stores vector embeddings and serves similarity search over indexed knowledge.
>
> It holds separate collections for **runtime knowledge bases** (queried by business agents at run time), **design-time knowledge bases** (used by the config-time agents in FlowX Designer), and the **AI assistant** corpus. Qdrant runs embedded as a single node by default; cluster mode is recommended for production.

***

## Platform data services

> [!NOTE]
>
> The following data services are required by the FlowX.AI platform but are third-party dependencies. You choose the deployment model (managed service or self-hosted, inside or outside the Kubernetes cluster) according to your enterprise standards. FlowX.AI connects to these services via configuration.
>
> For supported versions and compatibility details, see the [Third-party components](https://docs.flowx.ai/6.x/docs/platform-deep-dive/third-party-components.md#compatibility-matrix) compatibility matrix.

### Relational storage with PostgreSQL or Oracle

PostgreSQL is the primary relational database and system-of-record for most core services. Oracle Database is also supported as an alternative. They store:

* Process definitions and metadata
* Runtime instance state
* Platform configuration
* Administrative data

Most FlowX.AI microservices use PostgreSQL or Oracle as their authoritative storage layer.

### Document storage with MongoDB

MongoDB provides document-oriented storage for unstructured and semi-structured content. It stores:

* Runtime configuration
* Runtime workflow state
* Flexible data structures used by apps and integrations

MongoDB is also the storage layer for [FlowX Database](https://docs.flowx.ai/6.x/docs/platform-deep-dive/integrations/flowx-database.md), a cross-instance, long-term storage feature that lets apps persist and query data beyond a single process instance.

### Caching with Redis

Redis is an in-memory cache used for performance optimization across the platform. It stores:

* Cached process definitions
* Transient session data
* Distributed locks across microservices

> [!NOTE]
>
> Redis is used strictly for short-lived, performance-related data and is **not** a system-of-record. Long-term persistence is handled by PostgreSQL and MongoDB.

> [!NOTE]
>
> Message-start deduplication and pending external (received) messages are persisted in PostgreSQL, which is their source of truth. Message-start deduplication relies on a PostgreSQL unique constraint. Redis is read only as a fallback for pending messages written before the upgrade, so in-flight message processing survives a Redis restart or data loss.

### Event streaming with Kafka

Kafka is the event streaming backbone that enables asynchronous, event-driven communication between microservices. It handles:

* Internal event propagation between platform services
* External integration messaging
* Decoupled processing workflows

> [!NOTE]
>
> Kafka is used for transient message propagation. Authoritative business data is persisted in PostgreSQL and MongoDB, not in Kafka.

### S3-compatible object storage

S3-compatible object storage provides persistent file storage. It stores:

* File attachments
* Document outputs (generated PDFs, converted files)
* Large binary assets

FlowX.AI accesses object storage via an S3-compatible interface.

> [!TIP]
>
> For providers without native S3 support (such as Azure Blob Storage), an S3 proxy is required to expose an S3-compatible interface. If you use a native S3 service (like AWS S3), no proxy is needed.

> [!WARNING]
>
> On Azure Blob Storage, multipart uploads (files over 5 MB) through an S3 proxy can require Azure roles that include blob index tag permissions, which `Storage Blob Data Contributor` does not cover. See [Azure Blob Storage through an S3 proxy](https://docs.flowx.ai/6.x/setup-guides/documents-plugin-setup.md#azure-blob-storage-through-an-s3-proxy) for the required roles.

### Search and indexing with Elasticsearch

Elasticsearch is used primarily for audit logging and full-text search across workflow and runtime data. It stores:

* Searchable representations of audit events
* Platform activity logs
* Indexed workflow data for fast retrieval

Elasticsearch is optimized for fast search and traceability rather than primary data storage.

***

## AI Platform data stores

The AI Platform does not introduce a separate data stack. Its services reuse the same store families as the rest of the platform (PostgreSQL, MongoDB, Kafka, and object storage) and add **Qdrant** as the one AI-specific store for vector embeddings.

| Data store | How the AI Platform uses it |
| - | - |
| **Qdrant** | Vector embeddings and similarity search for knowledge-base RAG and AI agents. Separate collections for runtime knowledge bases, design-time knowledge bases, and the AI assistant. |
| **PostgreSQL** | Service state for AI components: embedding-job tracking and document-enrichment artifacts. Standalone services (document parser, speech-to-text) keep durable job state here. |
| **Kafka** | Asynchronous AI job topics (document parsing, speech-to-text, web crawling, evaluations, data privacy) and the internal knowledge-base pipeline that chains indexing, embedding, and enrichment. |
| **Object storage (S3)** | Knowledge-base enrichment offloads, parsed-document results, and agent-builder run artifacts. |
| **MongoDB** | AI Gateway conversation and message history (`ai-gateway` database). Knowledge-base **metadata** (store and entry definitions), owned by Integration Designer. The vectors themselves live in Qdrant, so a knowledge base spans MongoDB (metadata) and Qdrant (embeddings). |

> [!NOTE]
>
> All vector storage and similarity search across the AI Platform use **Qdrant**. PostgreSQL holds only service/job state for the AI components, not embeddings.

For exact databases, environment variables, Kafka topics, and provisioning details, see the [AI Platform setup](https://docs.flowx.ai/6.x/setup-guides/ai-platform-setup.md) guide.

***

## Quick reference

| Data Store | Role | What it stores | Persistence |
| - | - | - | - |
| **PostgreSQL / Oracle** | System of record | Process definitions, config, metadata, instance state | Permanent |
| **MongoDB** | Document store | Runtime state, flexible data, FlowX Database | Permanent |
| **Redis** | Cache | Process definitions cache, sessions, locks | Transient |
| **Kafka** | Event streaming | Inter-service messages, integration events | Transient |
| **Object Storage** | File store | Attachments, documents, binary assets | Permanent |
| **Elasticsearch** | Search engine | Audit logs, indexed workflow data | Permanent |
| **Qdrant** | Vector database (AI) | Embeddings for RAG and semantic retrieval | Permanent |

***

## Service-level database mapping

The tables and diagrams below show which FlowX.AI services connect to which databases and what operations they perform. This is useful for infrastructure sizing, backup planning, and troubleshooting.

### PostgreSQL databases

Each core service owns its own PostgreSQL database and manages its schema through Liquibase migrations at startup.

```mermaid theme={"dark"}
flowchart LR
    subgraph Services
        ADM["Admin"]
        ADV["Advancing Controller"]
        APPM["Application Manager"]
        AUTH["Authorization System"]
        DOC["Document Plugin"]
        EMAIL["Email Gateway"]
        INTD["Integration Designer"]
        LIC["License"]
        ORG["Organization Manager"]
        ENG["Process Engine"]
        TASK["Task Management"]
        DSYNC["Data Sync"]
        WHG["Webhook Gateway"]
    end

    subgraph PostgreSQL
        ADVDB[("advancing")]
        APPMDB[("app_manager")]
        AUTHDB[("authorization_system")]
        DOCDB[("document_plugin")]
        EMAILDB[("email_gateway")]
        LICDB[("license")]
        ORGDB[("org_manager")]
        ENGDB[("process_engine")]
        WHGDB[("webhook_gateway")]
    end

    ADM -->|"reads, writes"| ENGDB
    ADV -->|"creates, reads, writes"| ADVDB
    APPM -->|"creates, reads, writes"| APPMDB
    AUTH -->|"creates, reads, writes"| AUTHDB
    DOC -->|"creates, reads, writes"| DOCDB
    EMAIL -->|"creates, reads, writes"| EMAILDB
    INTD -->|"reads, writes"| ADVDB
    LIC -->|"creates, reads, writes"| LICDB
    ORG -->|"creates, reads, writes"| ORGDB
    ENG -->|"creates, reads, writes"| ENGDB
    ENG -->|"reads, writes"| ADVDB
    TASK -->|"reads, writes"| ENGDB
    WHG -->|"creates, reads, writes"| WHGDB
    DSYNC -.->|"manages migrations"| APPMDB
    DSYNC -.->|"manages migrations"| AUTHDB
    DSYNC -.->|"manages migrations"| ENGDB
```

| Service | Database | Operations |
| - | - | - |
| Admin | `process_engine` | Reads, writes |
| Advancing Controller | `advancing` | Creates, reads, writes |
| Application Manager | `app_manager` | Creates, reads, writes |
| Authorization System | `authorization_system` | Creates, reads, writes |
| Document Plugin | `document_plugin` | Creates, reads, writes |
| Email Gateway | `email_gateway` | Creates, reads, writes |
| Integration Designer | `advancing` | Reads, writes |
| License | `license` | Creates, reads, writes |
| Organization Manager | `org_manager` | Creates, reads, writes |
| Process Engine | `process_engine` | Creates, reads, writes |
| Process Engine | `advancing` | Reads, writes |
| Task Management | `process_engine` | Reads, writes |
| Webhook Gateway | `webhook_gateway` | Creates, reads, writes |

> [!NOTE]
>
> Admin and Task Management share the `process_engine` database with Process Engine. Process Engine and Integration Designer share the `advancing` database with Advancing Controller. Data Sync acts as a centralized migration coordinator for `app_manager`, `authorization_system`, and `process_engine`.

### MongoDB databases

Each service manages its own MongoDB database. The `app-runtime` database is a shared read-only dependency for multiple services.

```mermaid theme={"dark"}
flowchart LR
    subgraph Services
        ADM["Admin"]
        APPM["Application Manager"]
        RTM["Runtime Manager"]
        CMS["CMS Core"]
        DOC["Document Plugin"]
        INTD["Integration Designer"]
        NOTIF["Notification Plugin"]
        SCHED["Scheduler"]
        TASK["Task Management"]
        ENG["Process Engine"]
        EMAIL["Email Gateway"]
        FILEG["File Gateway"]
        AIGW["AI Gateway"]
        NOSQL["NoSQL DB Runner"]
    end

    subgraph MongoDB
        DM[("data-model")]
        AR[("app-runtime")]
        CMSDB[("cms-core")]
        DOCDB[("document-plugin")]
        NOTIFDB[("notification-plugin")]
        SCHEDDB[("scheduler-core")]
        TASKDB[("task-management-plugin")]
        INTDDB[("integration-designer")]
        NOSQLDB[("nosql-db-runner")]
        FILEGDB[("file-gateway")]
        AIGWDB[("ai-gateway")]
    end

    ADM -->|"creates, reads, writes"| DM
    APPM -->|"creates, reads, writes"| AR
    RTM -->|"reads, writes"| AR
    CMS -->|"creates, reads, writes"| CMSDB
    CMS -->|"reads"| AR
    DOC -->|"creates, reads, writes"| DOCDB
    DOC -->|"reads"| AR
    INTD -->|"creates, reads, writes"| INTDDB
    INTD -->|"reads"| AR
    NOTIF -->|"creates, reads, writes"| NOTIFDB
    NOTIF -->|"reads"| AR
    SCHED -->|"creates, reads, writes"| SCHEDDB
    TASK -->|"creates, reads, writes"| TASKDB
    TASK -->|"reads"| AR
    ENG -->|"reads"| AR
    EMAIL -->|"reads"| AR
    NOSQL -->|"creates, reads, writes"| NOSQLDB
    FILEG -->|"creates, reads, writes"| FILEGDB
    AIGW -->|"creates, reads, writes"| AIGWDB
```

| Service | Database | Operations |
| - | - | - |
| Admin | `data-model` | Creates, reads, writes |
| Application Manager | `app-runtime` | Creates, reads, writes |
| Runtime Manager | `app-runtime` | Reads, writes |
| CMS Core | `cms-core` | Creates, reads, writes |
| CMS Core | `app-runtime` | Reads |
| Document Plugin | `document-plugin` | Creates, reads, writes |
| Document Plugin | `app-runtime` | Reads |
| Integration Designer | `integration-designer` | Creates, reads, writes |
| Integration Designer | `app-runtime` | Reads |
| Notification Plugin | `notification-plugin` | Creates, reads, writes |
| Notification Plugin | `app-runtime` | Reads |
| Scheduler | `scheduler-core` | Creates, reads, writes |
| Task Management | `task-management-plugin` | Creates, reads, writes |
| Task Management | `app-runtime` | Reads |
| Process Engine | `app-runtime` | Reads |
| Email Gateway | `app-runtime` | Reads |
| NoSQL DB Runner | `nosql-db-runner` | Creates, reads, writes |
| File Gateway | `file-gateway` | Creates, reads, writes |
| AI Gateway | `ai-gateway` | Creates, reads, writes |

> [!WARNING]
>
> The `app-runtime` database is created by Application Manager, written by Runtime Manager when it deploys builds, and read by 7 other services. It contains runtime configuration, build data, and deployed app state. Availability of this database is critical for runtime operations.

### Elasticsearch indexes

| Service | Index | Operations |
| - | - | - |
| Audit Core | Audit index | Creates, reads, writes |
| Process Engine | Process instance index | Creates, reads, writes |
| Data Search | Process instance index | Reads |

### Redis

Core services use Redis as a cache via Spring Cache Manager. Events Gateway, Integration Designer, and Process Engine also use Redis for more than caching, as the following table shows.

Services using Redis for caching: Admin, Application Manager, Authorization System, CMS Core, Document Plugin, Email Gateway, Integration Designer, License, Notification Plugin, Organization Manager, Process Engine, Runtime Manager, Task Management.

| Service | Usage |
| - | - |
| Events Gateway | Cache, Redis Streams to route messages to the instance that holds the SSE connection |
| Integration Designer | Cache, pub/sub for workflow cancellation |
| Process Engine | Cache, pub/sub for runtime configuration parameter changes |

### S3-compatible object storage buckets

> [!NOTE]
>
> Bucket names shown are defaults from the configuration and are configurable via environment variables (for example, `MINIO_BUCKET_PREFIX`). Your deployment may use different names. In some deployments, multiple services may share the same bucket. Ensure cross-bucket read access is configured where needed.
>
> The bucket prefix value (`MINIO_BUCKET_PREFIX` for Document Plugin, `APPLICATION_FILESTORAGE_S3_BUCKETPREFIX` for Notification Plugin) is a **prefix**, not a single bucket name. The service derives per-type buckets from it as `<prefix>-<type>`. For example, the Document Plugin creates `flowx-dev-bucket-test-documents` for test files. The table shows the default prefix.

| Service | Default Bucket | Env Variable | Purpose |
| - | - | - | - |
| Application Manager | `applications-bucket` | — | App builds and exported/imported resources |
| CMS Core | `media-library-bucket` | — | Public media assets |
| CMS Core | `cms-private-storage-bucket` | — | Private theme and font files |
| Document Plugin | `flowx-dev-bucket` | `MINIO_BUCKET_PREFIX` | Document templates and generated files |
| Document Plugin | `temp-bucket` | `MINIO_TEMP_BUCKET` | Temporary processing files |
| Integration Designer | `workflows-bucket` | — | Workflow definitions and test files |
| Notification Plugin | `qdevlocal-preview-paperflow` | `APPLICATION_FILESTORAGE_S3_BUCKETPREFIX` | Notification templates and attachments |

### Qdrant collections

The AI Platform stores vector embeddings in separate Qdrant collections. The `embedder` service writes vectors; retrieval and design-time services read them.

| Collection | Written by | Read by | Content |
| - | - | - | - |
| Runtime knowledge bases | Embedder | Knowledgebase RAG, Planner | Embeddings for knowledge bases queried by business agents at run time |
| Design-time knowledge bases | Embedder | KB Enrichment, Planner, Knowledgebase RAG | Layout-aware embeddings for documents used by the config-time agents in FlowX Designer |
| AI assistant | Snapshot on startup | AI Assistant | Pre-indexed documentation corpus for the AI assistant |

> [!NOTE]
>
> Collection names are versioned and encode the embedding models in use, so exact names vary by deployment and release. Treat the collection roles above as the stable reference.

***

## Related resources

> [card] [Third-party components](https://docs.flowx.ai/6.x/docs/platform-deep-dive/third-party-components.md)
>
> Supported versions and compatibility matrix

> [card] [FlowX Database](https://docs.flowx.ai/6.x/docs/platform-deep-dive/integrations/flowx-database.md)
>
> Cross-instance, long-term data storage using MongoDB

> [card] [FlowX.AI architecture](https://docs.flowx.ai/6.x/docs/platform-overview/flowx-architecture.md)
>
> Overall platform architecture and microservices overview

> [card] [Redis configuration](https://docs.flowx.ai/6.x/setup-guides/redis-configuration.md)
>
> Redis deployment modes and configuration

> [card] [AI Platform setup](https://docs.flowx.ai/6.x/setup-guides/ai-platform-setup.md)
>
> AI Platform data stores, environment variables, and Kafka topics


## Related topics

- [FlowX.AI architecture: services, layers, and data flow](https://docs.flowx.ai/6.x/docs/platform-overview/flowx-architecture.md)
- [Third-party components](https://docs.flowx.ai/6.x/docs/platform-deep-dive/third-party-components.md)
- [Migrate self-hosted FlowX.AI to the flowx-ai Helm chart](https://docs.flowx.ai/6.x/setup-guides/migrate-to-flowx-ai-helm-chart.md)
- [Security](https://docs.flowx.ai/6.x/docs/platform-overview/security.md)
- [Configure the flowx-ai Helm chart for your infrastructure](https://docs.flowx.ai/6.x/setup-guides/configure-flowx-ai-helm-chart.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.