> ## Documentation Index
> Fetch the complete documentation index at: https://docs.flowx.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Microservices setup guides

> Complete reference for deploying and configuring FlowX.AI microservices in your environment

FlowX.AI is built on a modern microservices architecture, allowing for scalable, resilient, and flexible deployments. Each microservice operates independently while collaborating with others to provide a complete enterprise solution.

## Deployment strategy

Deploying FlowX.AI microservices involves breaking down the application into modular components that can be independently deployed, scaled, and maintained. All microservices are delivered as Docker containers, making them suitable for deployment on any container orchestration platform, such as Kubernetes or Docker Swarm.

### Deployment prerequisites

Before beginning the deployment process, ensure you have:

* A Kubernetes cluster or Docker environment
* Access to a container registry
* Persistent storage for databases
* Network policies configured for inter-service communication
* Resource quotas and limits defined for each environment

### Recommended installation order

<Tip>
  Following the correct installation sequence is crucial for a successful deployment. This ordered approach prevents dependency issues and ensures each service has the required dependencies available when it initializes.
</Tip>

<Steps>
  <Step title="Infrastructure Components">
    First, deploy the foundational infrastructure:

    * **Databases**: PostgreSQL/Oracle for relational data, MongoDB for document storage
    * **Message Broker**: Kafka and ZooKeeper
    * **Caching**: Redis
    * **Identity Management**: Keycloak or other OAuth2 provider
  </Step>

  <Step title="Core Components">
    Next, deploy these core services in order:

    1. **Advancing Controller**: Manages process advancement and orchestration
    2. **Process Engine**: Handles business process execution and state management
  </Step>

  <Step title="Backend Services">
    Once the core components are operational, deploy these services (can be deployed in parallel):

    * **Admin Service**: Platform administration and configuration management
    * **Audit Core**: Compliance auditing and activity tracking
    * **Task Management**: Human task assignment and workflow
    * **Scheduler Core**: Job scheduling and time-based operations
    * **Data Search**: Indexing and searching capabilities
    * **Events Gateway**: Event routing and processing
    * **Document Plugin**: Document generation and management
    * **Notification Plugin**: Communication and alerts management
    * **Any additional plugins or extensions**
  </Step>

  <Step title="AI Platform Services">
    Deploy the AI Platform services for intelligent automation capabilities:

    The AI Platform is a set of Python services:

    * **Planner**: Intent understanding and task orchestration
    * **Agent Builder**: Configuration and runtime for low-code AI nodes
    * **Knowledgebase RAG**: Retrieval-augmented generation over the knowledge base
    * **Embedder**: Embedding generation
    * **Knowledgebase Indexer v2**: Event-driven retrieval-vector indexing
    * **AI Developer**: Automated code generation capabilities
    * **AI Analyst**: Process analysis and optimization
    * **AI Designer**: UI generation and design automation
    * **AI Assistant**: FlowX.AI documentation assistant

    <Note>
      The AI Platform services are optional but provide significant value for organizations looking to use AI capabilities in their business processes.
    </Note>
  </Step>

  <Step title="Frontend Components">
    Finally, deploy the frontend services:

    * **Designer**: Process design environment for business analysts
    * **Web Components**: UI components for custom applications
    * **Customer-facing UIs and portals**
  </Step>
</Steps>

## Environment variables reference

Environment variables are the primary configuration mechanism for FlowX.AI microservices. They provide a secure and flexible way to customize service behavior without modifying the container images.

The following sections detail the most commonly used environment variables across FlowX.AI microservices. For service-specific variables, refer to the dedicated setup guides for each component.

<Warning>
  **A configuration that blocks startup is a required configuration.** If a service fails to start and its logs report a missing or empty configuration value, treat that value as required and provide it, even when it is not flagged as required here. These guides document the most common variables; they are not an exhaustive list of every value your deployment topology requires.
</Warning>

### Authorization & access management

| Environment Variable                                | Description                                        | Example Value                       | Required |
| --------------------------------------------------- | -------------------------------------------------- | ----------------------------------- | -------- |
| `SECURITY_OAUTH2_BASESERVERURL`                     | Base URL of the OAuth2/OIDC server                 | `https://keycloak.example.com/auth` | Yes      |
| `SECURITY_OAUTH2_CLIENTCLIENTID`                    | OAuth2 client ID for the service                   | `flowx-admin-service`               | Yes      |
| `SECURITY_OAUTH2_CLIENTCLIENTSECRET`                | OAuth2 client secret                               | `secret`                            | Yes      |
| `SECURITY_OAUTH2_REALM`                             | OAuth2 realm name                                  | `flowx`                             | Yes      |
| `SECURITY_OAUTH2_SERVICEACCOUNT_ADMIN_CLIENTID`     | Service account client ID (for inter-service auth) | `flowx-service-account`             | No\*     |
| `SECURITY_OAUTH2_SERVICEACCOUNT_ADMIN_CLIENTSECRET` | Service account client secret                      | `secret`                            | No\*     |

\*Required for some services that need to make authenticated calls to other services

### Database configuration

#### PostgreSQL/Oracle

| Environment Variable                            | Description                     | Example Value                           | Required |
| ----------------------------------------------- | ------------------------------- | --------------------------------------- | -------- |
| `SPRING_DATASOURCE_URL`                         | JDBC connection URL             | `jdbc:postgresql://postgres:5432/flowx` | Yes      |
| `SPRING_DATASOURCE_USERNAME`                    | Database username               | `flowx_user`                            | Yes      |
| `SPRING_DATASOURCE_PASSWORD`                    | Database password               | `securePassword123`                     | Yes      |
| `SPRING_DATASOURCE_DRIVERCLASSNAME`             | JDBC driver class (Oracle only) | `oracle.jdbc.OracleDriver`              | Yes\*    |
| `SPRING_JPA_PROPERTIES_HIBERNATE_DEFAULTSCHEMA` | Default schema (Oracle only)    | `FLOWX`                                 | Yes\*    |

\*Required only for Oracle databases

#### MongoDB (NoSQL)

| Environment Variable                  | Description                                | Example Value                                                               | Required |
| ------------------------------------- | ------------------------------------------ | --------------------------------------------------------------------------- | -------- |
| `SPRING_DATA_MONGODB_URI`             | MongoDB connection URI                     | `mongodb://mongo1,mongo2,mongo3:27017/flowx?replicaSet=rs0`                 | Yes      |
| `DB_USERNAME`                         | MongoDB username for services that need it | `flowx_mongo_user`                                                          | Yes      |
| `DB_PASSWORD`                         | MongoDB password                           | `mongoSecurePass456`                                                        | Yes      |
| `SPRING_DATA_MONGODB_RUNTIME_ENABLED` | Enable runtime MongoDB connection          | `true`                                                                      | No       |
| `SPRING_DATA_MONGODB_RUNTIME_URI`     | URI for runtime MongoDB connection         | `mongodb://${DB_USERNAME}:${DB_PASSWORD}@mongodb-runtime:27017/app-runtime` | No\*     |

\*Required if runtime MongoDB connection is enabled

### Kafka configuration

| Environment Variable               | Description                                       | Example Value                            | Required |
| ---------------------------------- | ------------------------------------------------- | ---------------------------------------- | -------- |
| `KAFKA_BOOTSTRAP_SERVERS`          | Comma-separated list of Kafka brokers (preferred) | `kafka-0:9092,kafka-1:9092,kafka-2:9092` | Yes\*    |
| `SPRING_KAFKA_BOOTSTRAPSERVERS`    | Legacy property for Kafka brokers (fallback)      | `kafka-0:9092,kafka-1:9092,kafka-2:9092` | Yes\*    |
| `KAFKA_SECURITY_PROTOCOL`          | Security protocol for Kafka (preferred)           | `SASL_PLAINTEXT`                         | No       |
| `SPRING_KAFKA_SECURITY_PROTOCOL`   | Legacy property for security protocol (fallback)  | `SASL_PLAINTEXT`                         | No       |
| `SPRING_KAFKA_CONSUMER_GROUPID`    | Consumer group ID                                 | `flowx-process-engine-group`             | Yes      |
| `KAFKA_CONSUMER_THREADS`           | Number of Kafka consumer threads                  | `5`                                      | No       |
| `KAFKA_AUTHEXCEPTIONRETRYINTERVAL` | Retry interval for auth exceptions (seconds)      | `10`                                     | No       |
| `KAFKA_MESSAGE_MAX_BYTES`          | Maximum Kafka message size (bytes)                | `52428800` (50 MB)                       | No       |

<Info>
  The `KAFKA_BOOTSTRAP_SERVERS` and `KAFKA_SECURITY_PROTOCOL` properties are preferred and aligned across all services. The legacy `SPRING_KAFKA_BOOTSTRAPSERVERS` and `SPRING_KAFKA_SECURITY_PROTOCOL` properties remain supported for backwards compatibility. If both are set, the new properties take precedence.

  \*At least one of `KAFKA_BOOTSTRAP_SERVERS` or `SPRING_KAFKA_BOOTSTRAPSERVERS` must be set.
</Info>

#### Kafka authentication profile

FlowX.AI supports enabling Kafka authentication across all Java microservices using the `kafka-auth` Spring configuration profile. This feature simplifies the activation of secure Kafka communication.

To enable Kafka authentication, set the `SPRING_PROFILES_ACTIVE` environment variable:

```bash theme={"system"}
SPRING_PROFILES_ACTIVE=kafka-auth
```

<Info>
  You can activate multiple profiles by separating them with commas: `SPRING_PROFILES_ACTIVE=dev,kafka-auth`
</Info>

**Additional environment variables for OAuth-based Kafka authentication:**

| Environment Variable             | Description                                                    | Example Value                             | Required |
| -------------------------------- | -------------------------------------------------------------- | ----------------------------------------- | -------- |
| `KAFKA_OAUTH_CLIENT_ID`          | OAuth client ID used to authenticate with the token endpoint   | `kafka-client`                            | Yes\*    |
| `KAFKA_OAUTH_CLIENT_SECRET`      | Secret associated with the OAuth client ID                     | `kafka-secret`                            | Yes\*    |
| `KAFKA_OAUTH_TOKEN_ENDPOINT_URI` | OAuth token endpoint URI from which access tokens are obtained | `https://keycloak.example.com/auth/token` | Yes\*    |

\*Required when `kafka-auth` profile is active

<Warning>
  The kafka-auth profile enforces SASL/OAUTHBEARER with plaintext transport. Ensure secure networking (VPN, mTLS) is configured where required.
</Warning>

For more details, see the [Kafka Authentication Configuration](./kafka-authentication-config) guide.

### Redis configuration

FlowX.AI microservices use Redis for caching and real-time data management. Redis can be deployed in three modes: Standalone, Sentinel (high availability), and Cluster (horizontal scaling).

**Quick reference for common Redis variables:**

| Environment Variable                | Description                   | Example Value             | Mode       |
| ----------------------------------- | ----------------------------- | ------------------------- | ---------- |
| `SPRING_DATA_REDIS_HOST`            | Redis server hostname         | `localhost`               | Standalone |
| `SPRING_DATA_REDIS_PORT`            | Redis server port             | `6379`                    | Standalone |
| `SPRING_DATA_REDIS_PASSWORD`        | Redis authentication password | `yourpassword`            | All modes  |
| `SPRING_DATA_REDIS_SENTINEL_MASTER` | Sentinel master name          | `mymaster`                | Sentinel   |
| `SPRING_DATA_REDIS_SENTINEL_NODES`  | Sentinel nodes list           | `host1:26379,host2:26379` | Sentinel   |
| `SPRING_DATA_REDIS_CLUSTER_NODES`   | Cluster nodes list            | `host1:6379,host2:6379`   | Cluster    |

<Info>
  For complete Redis configuration including all deployment modes, SSL/TLS setup, and troubleshooting, see the [Redis Configuration](./redis-configuration) guide.
</Info>

### Logging configuration

| Environment Variable         | Description                        | Default Value | Required |
| ---------------------------- | ---------------------------------- | ------------- | -------- |
| `LOGGING_LEVEL_ROOT`         | Root logging level                 | `INFO`        | No       |
| `LOGGING_LEVEL_APP`          | Application-specific logging level | `DEBUG`       | No       |
| `LOGGING_LEVEL_MONGO_DRIVER` | MongoDB driver logging level       | `WARN`        | No       |
| `LOGGING_LEVEL_KAFKA`        | Kafka client logging level         | `WARN`        | No       |
| `LOGGING_LEVEL_REDIS`        | Redis client logging level         | `OFF`         | No       |

#### Changing log levels at runtime

All FlowX microservices support dynamic log level adjustment without requiring a restart. This capability uses Spring Boot Actuator's loggers endpoint and is particularly useful for troubleshooting production issues.

**Prerequisites:**

Ensure the `loggers` endpoint is exposed in your management endpoints configuration:

**Example usage:**

```bash theme={"system"}
curl 'http://localhost:8081/manage/actuator/loggers/ai.flowx' \
  -i -X POST \
  -H 'Content-Type: application/json' \
  -d '{"configuredLevel":"DEBUG"}'
```

Replace `ai.flowx` with the specific logger package you want to modify.

**Available log levels:** `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, `OFF`

<Warning>
  Dynamic log level changes are not persisted. They will reset to the configured values when the service restarts. Using `DEBUG` or `TRACE` levels in production may impact performance and generate large log volumes.
</Warning>

## Deployment best practices

### High availability considerations

For production environments, configure these high availability features:

* **Database Clustering**: Implement PostgreSQL/Oracle with replication
* **MongoDB Replica Sets**: Deploy MongoDB as a replica set with at least 3 nodes
* **Kafka Clustering**: Use at least 3 Kafka brokers with replication factor ≥ 3
* **Redis Sentinel/Cluster**: Configure Redis for high availability
* **Service Replicas**: Run multiple instances of each microservice
* **Load Balancing**: Implement proper load balancing for service instances
* **Affinity/Anti-Affinity Rules**: Distribute service instances across nodes

### Security recommendations

Secure your FlowX.AI deployment with these measures:

1. **Network Segmentation**: Isolate microservices using network policies
2. **Secret Management**: Use Kubernetes Secrets or a vault solution
3. **TLS Everywhere**: Enable TLS for all service-to-service communication
4. **OAuth2 Scopes**: Configure fine-grained OAuth2 scopes for services
5. **Resource Isolation**: Use namespaces and pod security policies
6. **Regular Updates**: Keep all components updated with security patches
7. **Audit Logging**: Enable comprehensive audit logging via the Audit Core service

## Troubleshooting

### Common issues and solutions

| Issue                      | Possible cause                | Solution                                                        |
| -------------------------- | ----------------------------- | --------------------------------------------------------------- |
| Service fails to start     | Missing environment variables | Check logs for "Configuration property not found" errors        |
| Database connection errors | Incorrect credentials or URL  | Verify database connection parameters                           |
| Services can't communicate | Kafka misconfiguration        | Ensure topic names match between producer and consumer services |
| Authentication failures    | OAuth2 configuration issues   | Verify client IDs, secrets, and server URLs                     |
| Performance degradation    | Insufficient resources        | Monitor CPU/memory usage and scale resources appropriately      |
| Data inconsistency         | Redis cache not synchronized  | Check Redis connection and consider cache invalidation          |

### Diagnostic procedures

When troubleshooting FlowX.AI microservices:

1. **Check Service Logs**: Examine logs for error messages
2. **Verify Configurations**: Ensure all required environment variables are set correctly
3. **Test Connectivity**: Verify network connectivity between services
4. **Monitor Resources**: Check CPU, memory, and disk usage
5. **Inspect Kafka Topics**: Use Kafka tools to inspect message flow
6. **Review Database State**: Examine database for data integrity issues
7. **Check OAuth2 Tokens**: Verify token validity and permissions

## Service-specific documentation

For detailed configuration of individual services, refer to:

### Core platform components

<CardGroup cols={2}>
  <Card title="Process Engine Setup" href="../setup-guides/flowx-engine-setup-guide/engine-setup" icon="gear" />

  <Card title="Admin Setup" href="../setup-guides/admin-setup-guide" icon="shield" />

  <Card title="Advancing Controller Setup" href="../setup-guides/advancing-controller-setup-guide" icon="gear" />

  <Card title="Application Manager Setup" href="../setup-guides/application-manager" icon="gear" />

  <Card title="Runtime Manager Setup" href="../setup-guides/runtime-manager" icon="database" />

  <Card title="Scheduler Core Setup" href="../setup-guides/scheduler-setup-guide" icon="clock" />

  <Card title="Audit Setup" href="../setup-guides/audit-setup-guide" icon="clipboard-list" />

  <Card title="Authorization Setup" href="../setup-guides/authorization-setup-guide" icon="key" />

  <Card title="Task Management Setup" href="../setup-guides/task-management-plugin-setup" icon="list-check" />

  <Card title="Notifications Setup" href="../setup-guides/notifications-plugin-setup" icon="envelope" />

  <Card title="Document Setup" href="../setup-guides/documents-plugin-setup" icon="file" />

  <Card title="Events Gateway Setup" href="../setup-guides/events-gateway-setup" icon="envelope" />
</CardGroup>

### AI Platform services

<Info>
  The AI Platform is a suite of AI services that provide intelligent automation capabilities across the entire application lifecycle.
</Info>

<Card title="AI Platform Setup" href="../ai-platform/ai-platform-setup" icon="robot" />

### Plugins & extensions

<CardGroup cols={2}>
  <Card title="OCR Plugin Setup" href="../setup-guides/plugins-setup-guide/ocr-plugin-setup" icon="scan-line" />

  <Card title="Reporting Setup" href="../setup-guides/plugins-setup-guide/reporting-setup" icon="chart-bar" />
</CardGroup>

### Integration & data services

<CardGroup cols={2}>
  <Card title="Integration Designer Setup" href="../setup-guides/integration-designer-setup" icon="puzzle-piece" />

  <Card title="Data Search Setup" href="../setup-guides/search-data-service-setup-guide" icon="magnifying-glass" />

  <Card title="Data Sync Setup" href="../setup-guides/data-sync" icon="arrows-rotate" />

  <Card title="NoSQL DB Runner Setup" href="../setup-guides/nosql-db-runner" icon="database" />
</CardGroup>

### User interface & designer

<CardGroup cols={2}>
  <Card title="Designer Setup" href="../setup-guides/designer-setup-guide" icon="palette" />

  <Card title="CMS Setup" href="../setup-guides/cms-setup" icon="file" />
</CardGroup>

### Infrastructure & extensions

<CardGroup cols={2}>
  <Card title="SpiceDB Setup" href="../setup-guides/spicedb" icon="shield-check" />
</CardGroup>

### Observability & configuration

<CardGroup cols={2}>
  <Card title="OpenTelemetry Config" href="../setup-guides/open-telemetry-config" icon="chart-line" />

  <Card title="OT Default Properties" href="../setup-guides/ot-default-properties" icon="scroll" />
</CardGroup>