> ## Documentation Index > Fetch the complete documentation index at: https://docs.flowx.ai/llms.txt > Use this file to discover all available pages before exploring further. # FlowX Engine setup > This guide provides instructions on how to set up and configure the FlowX.AI Engine. ## Infrastructure prerequisites Before installing the FlowX.AI Engine, verify that the following infrastructure components are installed and configured: * **Kafka** * **Elasticsearch** * **PostgreSQL** or **Oracle** * **MongoDB** *** ## Dependencies The FlowX Engine requires the following components: * **Database**: Primary storage for the engine (PostgreSQL or Oracle) * **Redis Server**: Used for caching. See [Redis Configuration](#redis-configuration) * **Kafka**: Handles messaging and event-driven architecture. See [Configuring Kafka](#configuring-kafka) For a microservices architecture, services typically manage their data via dedicated databases. ### Required external services * **Redis Cluster**: Caches process definitions, compiled scripts, and Kafka responses * **Kafka Cluster**: Enables communication with external plugins and integrations *** ## Configuration setup FlowX.AI Engine uses environment variables for configuration. This section covers key configuration areas: * [Database configuration](#database-configuration) * [Authorization & access roles](#authorization--access-roles) * [Kafka configuration](#configuring-kafka) * [File upload size](#configuring-file-upload-size) * [Elasticsearch connection](#configuring-elasticsearch-connection) ### Database configuration The FlowX Engine supports both PostgreSQL and Oracle as its primary relational database. #### PostgreSQL | Environment variable | Description | | ----------------------------------- | --------------------------- | | `SPRING_DATASOURCE_URL` | Database URL for PostgreSQL | | `SPRING_DATASOURCE_USERNAME` | Username for PostgreSQL | | `SPRING_DATASOURCE_PASSWORD` | Password for PostgreSQL | | `SPRING_DATASOURCE_DRIVERCLASSNAME` | Driver class for PostgreSQL | #### Oracle | Environment variable | Description | | ----------------------------------- | ----------------------- | | `SPRING_DATASOURCE_URL` | Database URL for Oracle | | `SPRING_DATASOURCE_USERNAME` | Username for Oracle | | `SPRING_DATASOURCE_PASSWORD` | Password for Oracle | | `SPRING_DATASOURCE_DRIVERCLASSNAME` | Driver class for Oracle | **Default:** `oracle.jdbc.OracleDriver` #### MongoDB configuration Configure connection to the Runtime MongoDB instance: | Environment variable | Description | Default value | | --------------------------------- | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | | `SPRING_DATA_MONGODB_RUNTIME_URI` | URI for connecting to MongoDB | Format: `mongodb://${RUNTIME_DB_USERNAME}:${DB_PASSWORD}@,,:/${DB_NAME}?retryWrites=false` | | `RUNTIME_DB_USERNAME` | MongoDB username | `app-runtime` | | `DB_NAME` | MongoDB database name | `app-runtime` | ### Configuration parameters There are two types of Config Params that can be read from the environment: **variables** and **secrets**. There is one provider for variables and secrets extracted from the environment variables, and two providers for the ones extracted from Kubernetes. By default, the variables and secrets are extracted from environment variables (`env` provider). #### Configuration parameters from environment variables (default) The `env` provider used for variables and secrets extracts them from environment variables. For security reasons, the `env` provider uses an allow list regex which defaults to `FLOWX_CONFIGPARAM_.*`. This means only environment variables that match this naming pattern can be read at runtime into configuration params (either as variables or secrets). Feel free to edit it to match the environment variables that you use in your deployment. | Environment variable | Description | Default value | | ------------------------------------------- | --------------------------------------------------------------- | ---------------------- | | `FLOWX_CONFIGPARAMS_VARS_PROVIDER` | Provider type for variables | `env` | | `FLOWX_CONFIGPARAMS_VARS_ALLOWLISTREGEX` | Regular expression to match allowed env variables for variables | `FLOWX_CONFIGPARAM_.*` | | `FLOWX_CONFIGPARAMS_SECRETS_PROVIDER` | Provider type for secrets | `env` | | `FLOWX_CONFIGPARAMS_SECRETS_ALLOWLISTREGEX` | Regular expression to match allowed env variables for secrets | `FLOWX_CONFIGPARAM_.*` | #### Configuration parameters from Kubernetes Secrets and ConfigMaps Use the following configuration to read Config Params from Kubernetes Secrets and ConfigMaps: | Environment variable | Description | Values | | ------------------------------------- | --------------------------- | ---------------- | | `FLOWX_CONFIGPARAMS_VARS_PROVIDER` | Provider type for variables | `k8s-configmaps` | | `FLOWX_CONFIGPARAMS_SECRETS_PROVIDER` | Provider type for secrets | `k8s-secrets` | These providers can be configured as follows: | Environment variable | Description | Values | | -------------------------------------------------------------- | ------------------------------------------ | -------------------- | | `FLOWX_CONFIGPARAMS_PROVIDERS_K8SCONFIGMAPS_CONFIGMAPSLIST_0_` | Name of the ConfigMap to use for variables | `flowx-configparams` | | `FLOWX_CONFIGPARAMS_PROVIDERS_K8SSECRETS_SECRETSLIST_0_` | Name of the Secret to use for secrets | `flowx-configparams` | You can configure multiple secrets and ConfigMaps by incrementing the index number (e.g., `FLOWX_CONFIGPARAMS_PROVIDERS_K8SSECRETS_SECRETSLIST_1`, `FLOWX_CONFIGPARAMS_PROVIDERS_K8SCONFIGMAPS_CONFIGMAPSLIST_1`). Values are overridden based on the order in which the maps are defined. The default provider is `env`, but there is a built-in allowlist with the regex pattern `FLOWX_CONFIGPARAM_.*`. This means only configuration parameters that match this naming pattern can be read at runtime, whether they are environment variables or secret variables. #### Config params cache Configure caching behavior for configuration parameters to optimize performance: | Environment variable | Description | Default value | | ------------------------------------- | ------------------------------------------------------- | ------------- | | `FLOWX_CACHE_CONFIGPARAMS_TTLSECONDS` | Time-to-live in seconds for cached configuration params | `120` | | `FLOWX_CACHE_CONFIGPARAMS_MAXENTRIES` | Maximum number of entries in the config params cache | `500` | These settings control how long configuration parameters are cached in memory before being refreshed from the source (environment variables, ConfigMaps, or Secrets). Adjust these values based on how frequently your configuration changes and your memory constraints. ### Authorization & access roles This section covers OAuth2 configuration settings for securing the Spring application. #### Security configuration | Environment variable | Description | Default value | | -------------------------------------------- | ------------------------------------------------------ | ----------------------------------- | | `SECURITY_TYPE` | Token validation mechanism (JWT public key validation) | `jwt-public-key` | | `SECURITY_BASIC_ENABLED` | Enable basic authentication | `false` | | `SECURITY_PUBLICPATHS_0` | Public path not requiring authentication | `/api/platform/components-versions` | | `SECURITY_PUBLICPATHS_1` | Public path not requiring authentication | `/manage/actuator/health` | | `SECURITY_PATHAUTHORIZATIONS_0_PATH` | Security path pattern | `"/api/**"` | | `SECURITY_PATHAUTHORIZATIONS_0_ROLESALLOWED` | Roles allowed for path access | `"ANY_AUTHENTICATED_USER"` | #### Service-to-service authentication The engine validates incoming tokens with the JWT public key mechanism and authenticates to other FlowX services with a dedicated service account (the `mainIdentity` client registration) in the service-accounts realm: | Environment variable | Description | Default value | | --------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | `SECURITY_OAUTH2_BASESERVERURL` | Base URL of the Keycloak server | | | `SECURITY_OAUTH2_SAREALM` | Service-accounts realm ID | `00000002-0002-4002-8002-000000000002` | | `FLOWX_LIB_SECURITY_SERVICES_ORGANIZATIONMANAGER_BASEURL` | URL of the organization-manager service, used by the security library | `http://organization-manager:80` | | `SPRING_SECURITY_OAUTH2_CLIENT_REGISTRATION_MAINIDENTITY_CLIENTID` | Service account client ID | `flowx-process-engine-sa` | | `SPRING_SECURITY_OAUTH2_CLIENT_REGISTRATION_MAINIDENTITY_CLIENTSECRET` | Service account client secret (Keycloak-issued) | | | `SPRING_SECURITY_OAUTH2_CLIENT_REGISTRATION_ANONYMOUSIDENTITY_CLIENTID` | Anonymous service account client ID, used for [anonymous runtime access](/5.9/docs/platform-deep-dive/user-roles-management/anonymous-runtime-access) | `flowx-anonymous-sa` | | `SPRING_SECURITY_OAUTH2_CLIENT_REGISTRATION_ANONYMOUSIDENTITY_CLIENTSECRET` | Anonymous service account client secret | | | `SPRING_SECURITY_OAUTH2_CLIENT_PROVIDER_MAINAUTHPROVIDER_TOKENURI` | Provider token URI, resolved against the service-accounts realm | `${SECURITY_OAUTH2_BASESERVERURL}/realms/${SECURITY_OAUTH2_SAREALM}/protocol/openid-connect/token` | **Upgrading from 5.1.x?** Remove the legacy opaque-token env vars: `SECURITY_OAUTH2_REALM`, `SECURITY_OAUTH2_CLIENT_CLIENTID`, `SECURITY_OAUTH2_CLIENT_CLIENTSECRET`, and `SECURITY_OAUTH2_SERVICEACCOUNT_ADMIN_*`. These belong to the removed introspection model and prevent the service from starting on 5.9.x. See the [authentication and IAM migration guide](/5.9/migrating-from-5.1-lts/authentication-iam) for the full list. When deploying with the FlowX Helm chart, `SECURITY_OAUTH2_BASESERVERURL` is supplied through the chart value `flowx.keycloak.baseServerUrl` and the service-account client secrets are injected from the chart-managed Keycloak secret. The remaining values ship as image defaults. For more information about the necessary service account, see [Process Engine Service Account](../access-management/configuring-an-iam-solution#process-engine-service-account). ### Configuring Kafka Kafka handles all communication between the FlowX.AI Engine, external plugins, and integrations. It also notifies running process instances when certain events occur. #### Kafka connection settings | Environment variable | Description | Default value | | ------------------------- | ------------------------------------------------------------------------ | ---------------- | | `KAFKA_BOOTSTRAP_SERVERS` | Kafka bootstrap servers (fallback: `SPRING_KAFKA_BOOTSTRAP_SERVERS`) | `localhost:9092` | | `KAFKA_SECURITY_PROTOCOL` | Security protocol for Kafka (fallback: `SPRING_KAFKA_SECURITY_PROTOCOL`) | `PLAINTEXT` | #### Message routing configuration | Environment variable | Description | Default value | | -------------------------- | --------------------------------------------------------------------- | ------------------- | | `KAFKA_DEFAULT_FX_CONTEXT` | Default context value for message routing when no context is provided | `""` (empty string) | When `KAFKA_DEFAULT_FX_CONTEXT` is set and an event is received on Kafka without an fxContext header, the system will automatically apply the default context value to the message. #### Kafka consumer retry settings | Environment variable | Description | Default value | | ---------------------------------- | ----------------------------------------------------------------- | ------------- | | `KAFKA_AUTHEXCEPTIONRETRYINTERVAL` | Interval between retries after `AuthorizationException` (seconds) | `10` | #### Consumer groups & consumer threads configuration Both a producer and a consumer must be configured: ![FlowX Engine Kafka communication pattern](https://s3.eu-west-1.amazonaws.com/docx.flowx.ai/platform-deep-dive/engine_kafka_pattern.svg) ##### About consumer groups and threads A consumer group is a set of consumers that jointly consume messages from one or more Kafka topics. Each consumer group has a unique identifier (group ID) that Kafka uses to manage message distribution. Thread numbers refer to the number of threads a consumer application uses to process messages. Increasing thread count can improve parallelism and efficiency, especially with high message volumes. ##### Consumer group configuration | Environment variable | Description | Default value | | ------------------------------------------------ | ------------------------------------------------------------------- | -------------------------- | | `KAFKA_CONSUMER_GROUPID_NOTIFYADVANCE` | Group ID for notifying advance actions | `advance` | | `KAFKA_CONSUMER_GROUPID_NOTIFYPARENT` | Group ID for notifying when a subprocess is blocked | `notify-parent` | | `KAFKA_CONSUMER_GROUPID_ADAPTERS` | Group ID for messages related to adapters | `adapters` | | `KAFKA_CONSUMER_GROUPID_SCHEDULERRUNACTION` | Group ID for running scheduled actions | `scheduler-run-action` | | `KAFKA_CONSUMER_GROUPID_SCHEDULERADVANCING` | Group ID for messages indicating continuing advancement | `scheduler-advancing` | | `KAFKA_CONSUMER_GROUPID_MESSAGEEVENTS` | Group ID for message events | `message-events` | | `KAFKA_CONSUMER_GROUPID_PROCESS_START` | Group ID for starting processes | `process-start` | | `KAFKA_CONSUMER_GROUPID_PROCESS_STARTFOREVENT` | Group ID for starting processes for an event | `process-start-for-event` | | `KAFKA_CONSUMER_GROUPID_PROCESS_EXPIRE` | Group ID for expiring processes | `process-expire` | | `KAFKA_CONSUMER_GROUPID_PROCESS_OPERATIONS` | Group ID for processing operations from Task Management plugin | `process-operations` | | `KAFKA_CONSUMER_GROUPID_PROCESS_BATCHPROCESSING` | Group ID for processing bulk operations from Task Management plugin | `process-batch-processing` | | `KAFKA_CONSUMER_GROUPID_PROCESS_UIFLOWUPDATE` | Group ID for processing UI flow session variable updates | `ui-flow-update` | | `KAFKA_CONSUMER_GROUPID_PROCESS_UIFLOWEXPIRE` | Group ID for processing UI flow session expiry triggers | `ui-flow-expire` | ##### Consumer thread configuration | Environment variable | Description | Default value | | ------------------------------------------------ | --------------------------------------------------------------------- | ------------- | | `KAFKA_CONSUMER_THREADS_NOTIFYADVANCE` | Number of threads for notifying advance actions | `6` | | `KAFKA_CONSUMER_THREADS_NOTIFYPARENT` | Number of threads for notifying when a subprocess is blocked | `6` | | `KAFKA_CONSUMER_THREADS_ADAPTERS` | Number of threads for processing messages related to adapters | `6` | | `KAFKA_CONSUMER_THREADS_SCHEDULERADVANCING` | Number of threads for continuing advancement | `6` | | `KAFKA_CONSUMER_THREADS_SCHEDULERRUNACTION` | Number of threads for running scheduled actions | `6` | | `KAFKA_CONSUMER_THREADS_MESSAGEEVENTS` | Number of threads for message events | `6` | | `KAFKA_CONSUMER_THREADS_PROCESS_START` | Number of threads for starting processes | `6` | | `KAFKA_CONSUMER_THREADS_PROCESS_STARTFOREVENT` | Number of threads for starting processes for an event | `2` | | `KAFKA_CONSUMER_THREADS_PROCESS_EXPIRE` | Number of threads for expiring processes | `6` | | `KAFKA_CONSUMER_THREADS_PROCESS_OPERATIONS` | Number of threads for processing operations from task management | `6` | | `KAFKA_CONSUMER_THREADS_PROCESS_BATCHPROCESSING` | Number of threads for processing bulk operations from task management | `6` | | `KAFKA_CONSUMER_THREADS_PROCESS_UIFLOWUPDATE` | Number of threads for processing UI flow session variable updates | `6` | | `KAFKA_CONSUMER_THREADS_PROCESS_UIFLOWEXPIRE` | Number of threads for processing UI flow session expiry triggers | `6` | All events that start with a configured pattern will be consumed by the Engine. This enables you to create new integrations and connect them to the engine without changing the configuration. #### Configuring Kafka topics ##### Topic naming configuration | Environment Variable | Description | Default Value | | ---------------------------------------------- | ----------------------------------- | ---------------------- | | `KAFKA_TOPIC_NAMING_PACKAGE` | Package prefix for topic names | `ai.flowx.` | | `KAFKA_TOPIC_NAMING_ENVIRONMENT` | Environment segment for topic names | ` ` | | `KAFKA_TOPIC_NAMING_VERSION` | Version suffix for topic names | `.v1` | | `KAFKA_TOPIC_NAMING_SEPARATOR` | Primary separator for topic names | `.` | | `KAFKA_TOPIC_NAMING_SEPARATOR2` | Secondary separator for topic names | `-` | | `KAFKA_TOPIC_NAMING_ENGINERECEIVEPATTERN` | Engine receive pattern | `engine.receive.` | | `KAFKA_TOPIC_NAMING_INTEGRATIONRECEIVEPATTERN` | Integration receive pattern | `integration.receive.` | ##### Core engine topics | Environment variable | Description | Default value | | ------------------------------------ | --------------------------------------------------------- | ----------------------------------------- | | `KAFKA_TOPIC_PROCESS_NOTIFY_ADVANCE` | Topic used internally for advancing processes | `ai.flowx.core.notify.advance.process.v1` | | `KAFKA_TOPIC_PROCESS_NOTIFY_PARENT` | Topic used for sub-processes to notify the parent process | `ai.flowx.core.notify.parent.process.v1` | | `KAFKA_TOPIC_PATTERN` | Pattern the Engine listens on for incoming events | `ai.flowx.engine.receive.*` | | `KAFKA_TOPIC_PROCESS_EVENT_MESSAGE` | Topic for process message events | `ai.flowx.core.message.event.process.v1` | ##### Topics related to the Task Management plugin | Environment variable | Description | Default value | | --------------------------------------- | ---------------------------------------------------------- | -------------------------------------------- | | `KAFKA_TOPIC_TASK_OUT` | Topic used for sending notifications to the plugin | `ai.flowx.plugin.tasks.trigger.save.task.v1` | | `KAFKA_TOPIC_PROCESS_OPERATIONS_IN` | Topic for receiving information about operations performed | `ai.flowx.core.trigger.operation.v1` | | `KAFKA_TOPIC_PROCESS_OPERATIONS_BULKIN` | Topic where operations can be performed in bulk | `ai.flowx.core.trigger.operations.bulk.v1` | ##### Topics related to the Notification plugin | Environment variable | Description | Default value | | ------------------------ | ------------------------------------------------------------------ | ----------------------------------------------------------- | | `KAFKA_NOTIFICATION_OUT` | Topic for sending notification requests to the Notification plugin | `ai.flowx.plugin.notification.trigger.send.notification.v1` | The `KAFKA_NOTIFICATION_OUT` topic must resolve to the same value as `KAFKA_TOPIC_NOTIFICATION_INTERNAL_IN` in the Notification plugin configuration. ##### Topics related to the Document plugin The process-engine receives results from document-plugin operations on topics matching the pattern `ai.flowx.engine.receive.plugin.document.*`. The following topics are used for file encryption and decryption operations: | Topic name | Description | | ----------------------------------------------------------------- | ---------------------------------------------------------------- | | `ai.flowx.engine.receive.plugin.document.encrypt.file.results.v1` | Topic for receiving file encryption results from document-plugin | | `ai.flowx.engine.receive.plugin.document.decrypt.file.results.v1` | Topic for receiving file decryption results from document-plugin | These topics are automatically matched by the Engine's `KAFKA_TOPIC_PATTERN` configuration (default: `ai.flowx.engine.receive.*`). No additional configuration is required. The topics must be created in your Kafka infrastructure before deployment. ###### OPERATIONS\_IN request example ```json theme={"system"} { "operationType": "UNASSIGN", //type of operation performed in Task Management plugin "taskId": "some task id", "processInstanceUuid": "1cff0b7d-966b-4b35-9e9b-63b1d6757ec6", "swimlaneName": "Default", "swimlaneId": "51ec1241-fe06-4576-9c84-31598c05c527", "owner": { "firstName": null, "lastName": null, "username": "service-account-flowx-process-engine-account", "enabled": false }, "author": "admin@flowx.ai" } ``` ###### BULK\_IN request example ```json theme={"system"} { "operations": [ { "operationType": "HOLD", "taskId": "some task id", "processInstanceUuid": "d3aabfd8-d041-4c62-892f-22d17923b223", // the id of the process instance "swimlaneName": "Default", //name of the swimlane "owner": null, "author": "john.doe@flowx.ai" }, { "operationType": "HOLD", "taskId": "some task id", "processInstanceUuid": "d3aabfd8-d041-4c62-892f-22d17923b223", "swimlaneName": "Default", //name of the swimlane "owner": null, "author": "john.doe@flowx.ai" } ] } ``` To send additional keys in the response, attach them in the header. For example, you can use a `requestID` key. A response should be sent on a `callbackTopic` if it is mentioned in the headers: ![Callback topic with requestID header example](https://s3.eu-west-1.amazonaws.com/docx.flowx.ai/platform-deep-dive/bulk_requestid.png) ```json theme={"system"} {"processInstanceId": ${processInstanceId}, "callbackTopic": "test.operations.out", "requestID":"1234567890"} ``` Process operations on this topic include: assignment, unassignment, hold, unhold, terminate. The Task Manager produces the assignment and hold operations on its `...operations.out` topic; terminate is available only to direct Kafka producers. For more information, see the Task Management plugin documentation: 📄 [**Task management plugin**](../../docs/platform-deep-dive/core-extensions/task-management/task-management-overview) ##### Topics related to UI flow session updates | Environment variable | Description | Default value | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------- | | `KAFKA_TOPIC_UIFLOW_UPDATE_IN` | Topic for receiving UI flow session variable updates | `ai.flowx.core.trigger.ui-flow.update.v1` | | `KAFKA_TOPIC_UIFLOW_EXPIRE_IN` | Topic for receiving UI flow session expiry triggers, published by scheduler-core when a session's expiry time is reached | `ai.flowx.core.trigger.expire.ui-flow.v1` | ##### Topics related to the scheduler extension | Environment variable | Description | Default value | | -------------------------------------------- | -------------------------------------------------------- | ------------------------------------------ | | `KAFKA_TOPIC_PROCESS_EXPIRE_IN` | Topic for requests to expire processes | `ai.flowx.core.trigger.expire.process.v1` | | `KAFKA_TOPIC_PROCESS_SCHEDULE_OUT_SET` | Topic used for scheduling process expiration | `ai.flowx.core.trigger.set.schedule.v1` | | `KAFKA_TOPIC_PROCESS_SCHEDULE_OUT_STOP` | Topic used for stopping process expiration | `ai.flowx.core.trigger.stop.schedule.v1` | | `KAFKA_TOPIC_PROCESS_SCHEDULE_IN_RUN_ACTION` | Topic for requests to run scheduled actions | `ai.flowx.core.trigger.run.action.v1` | | `KAFKA_TOPIC_PROCESS_SCHEDULE_IN_ADVANCE` | Topic for events related to advancing through a database | `ai.flowx.core.trigger.advance.process.v1` | ##### Topics related to Timer Events | Environment variable | Description | Default value | | --------------------------------------------------- | ----------------------------------------------- | ---------------------------------------------------- | | `KAFKA_TOPIC_PROCESS_SCHEDULEDTIMEREVENTS_OUT_SET` | Used to communicate with Scheduler microservice | `ai.flowx.core.trigger.set.timer-event-schedule.v1` | | `KAFKA_TOPIC_PROCESS_SCHEDULEDTIMEREVENTS_OUT_STOP` | Used to communicate with Scheduler microservice | `ai.flowx.core.trigger.stop.timer-event-schedule.v1` | ##### Topics related to the Search Data service | Environment variable | Description | Default value | | ----------------------------- | --------------------------------------------------------- | ----------------------------------------------------- | | `KAFKA_TOPIC_DATA_SEARCH_IN` | Topic that the Engine listens on for search requests | `ai.flowx.core.trigger.search.data.v1` | | `KAFKA_TOPIC_DATA_SEARCH_OUT` | Topic used by the Engine to reply after finding a process | `ai.flowx.engine.receive.core.search.data.results.v1` | ##### Topics related to the Audit service | Environment variable | Description | Default value | | ----------------------- | ---------------------------- | ------------------------------------- | | `KAFKA_TOPIC_AUDIT_OUT` | Topic for sending audit logs | `ai.flowx.core.trigger.save.audit.v1` | ##### Topics related to Elasticsearch indexing | Environment variable | Default value | | ------------------------------- | -------------------------------- | | `KAFKA_TOPIC_PROCESS_INDEX_OUT` | `ai.flowx.core.index.process.v1` | ##### Processes that can be started by sending messages to a Kafka topic | Environment variable | Description | Default value | | ------------------------------- | ----------------------------------------------------------------- | ---------------------------------------- | | `KAFKA_TOPIC_PROCESS_START_IN` | Topic for requests to start a new process instance | `ai.flowx.core.trigger.start.process.v1` | | `KAFKA_TOPIC_PROCESS_START_OUT` | Topic for sending the reply after starting a new process instance | `ai.flowx.core.confirm.start.process.v1` | ##### Topics related to Message Events | Environment variable | Default value | | ----------------------------------- | -------------------------------------------------- | | `KAFKA_TOPIC_PROCESS_STARTFOREVENT` | `ai.flowx.core.trigger.start-for-event.process.v1` | | | | ##### Topics related to Events-gateway microservice | Environment variable | Description | Default value | | ------------------------------------------ | --------------------------------------------------------- | ------------------------------------------------------ | | `KAFKA_TOPIC_EVENTSGATEWAY_OUT_MESSAGE` | Outgoing messages from process-engine to events-gateway | `ai.flowx.eventsgateway.engine.commands.message.v1` | | `KAFKA_TOPIC_EVENTSGATEWAY_OUT_DISCONNECT` | Disconnect commands from process-engine to events-gateway | `ai.flowx.eventsgateway.engine.commands.disconnect.v1` | | `KAFKA_TOPIC_EVENTSGATEWAY_OUT_CONNECT` | Connect commands from process-engine to events-gateway | `ai.flowx.eventsgateway.engine.commands.connect.v1` | ##### Topics related to platform components | Environment variable | Description | Default value | | ---------------------------------------------- | ---------------------------------- | ---------------------------------------------------- | | `KAFKA_TOPIC_PLATFORM_COMPONENTS_VERSIONS_OUT` | Topic for platform version caching | `ai.flowx.core.trigger.platform.versions.caching.v1` | ##### Inter-service topic coordination When configuring FlowX services, ensure the following: 1. The Engine's `pattern` must match the pattern used by services sending messages to the Engine 2. The `integrationPattern` must match the pattern used by the Integration Designer 3. Output topics from one service must match the expected input topics of another service For example: * Services send to topics matching `ai.flowx.engine.receive.*` → Engine listens * Engine sends to topics matching `ai.flowx.integration.receive.*` → Integration Designer listens ##### Kafka message size configuration | Environment variable | Description | Default value | | ------------------------- | ----------------------------- | ----------------- | | `KAFKA_MESSAGE_MAX_BYTES` | Maximum message size in bytes | `52428800` (50MB) | This setting affects: * Producer message max bytes * Producer max request size * Consumer max partition fetch bytes ##### Kafka authentication (when using SASL\_PLAINTEXT) For secure environments, enable OAuth authentication with the following environment variables: | Environment Variable | Description | Default Value | | -------------------------------- | -------------------- | ---------------------- | | `KAFKA_OAUTH_CLIENT_ID` | OAuth client ID | `kafka` | | `KAFKA_OAUTH_CLIENT_SECRET` | OAuth client secret | `kafka-secret` | | `KAFKA_OAUTH_TOKEN_ENDPOINT_URI` | OAuth token endpoint | `kafka.auth.localhost` | When using the `kafka-auth` profile, the security protocol will automatically be set to `SASL_PLAINTEXT` and the SASL mechanism will be set to `OAUTHBEARER`. ### Configuring Elasticsearch connection The Process Engine uses Elasticsearch for process instance indexing and search capabilities. Configure the connection using these environment variables: | Variable | Description | Default Value | | -------------------------------------- | ------------------------------------------- | ------------- | | `SPRING_ELASTICSEARCH_REST_PROTOCOL` | Connection protocol | `https` | | `SPRING_ELASTICSEARCH_REST_URIS` | URL(s) of Elasticsearch nodes (no protocol) | - | | `SPRING_ELASTICSEARCH_REST_DISABLESSL` | Disable SSL verification | `false` | | `SPRING_ELASTICSEARCH_REST_USERNAME` | Authentication username | - | | `SPRING_ELASTICSEARCH_REST_PASSWORD` | Authentication password | - | For indexing setup, check the [Configuring Elasticsearch indexing](./configuring-elasticsearch-indexing/elasticsearch-indexing) section. #### Indexing settings | Variable | Description | Default Value | | -------------------------------------------- | ------------------------------------------------- | ------------------ | | `FLOWX_INDEXING_ENABLED` | Enable/disable indexing globally | `true` | | `FLOWX_INDEXING_TYPE` | Indexing strategy (http, kafka, no-indexing) | `http` | | `FLOWX_ELASTICSEARCH_INDEXSETTINGS_NAME` | Name of the Elasticsearch index | `process_instance` | | `FLOWX_ELASTICSEARCH_INDEXSETTINGS_SHARDS` | Number of primary shards | `2` | | `FLOWX_ELASTICSEARCH_INDEXSETTINGS_REPLICAS` | Number of replica shards | `0` | | `FLOWX_INDEXING_OPTIMISTICLOCKINGRETRIES` | Retry attempts for optimistic locking (HTTP only) | `3` | ### Configuring file upload size | Environment variable | Description | Default value | | ----------------------------------------- | ---------------------------------------- | ------------- | | `SPRING_SERVLET_MULTIPART_MAXFILESIZE` | Maximum file size allowed for uploads | `50MB` | | `SPRING_SERVLET_MULTIPART_MAXREQUESTSIZE` | Maximum request size allowed for uploads | `50MB` | #### Connecting the Advancing controller To use the advancing controller, configure the following variables: | Environment variable | Description | Default value | | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------- | | `ADVANCING_DATASOURCE_URL` | Connection URL for Advancing Postgres DB | `jdbc:postgresql://postgresql:5432/advancing` | | `ADVANCING_DATASOURCE_USERNAME` | Username for Advancing DB connection | `flowx` | | `ADVANCING_DATASOURCE_PASSWORD` | Password for Advancing DB connection | `-` | | `ADVANCING_DATASOURCE_MAXIMUMPOOLSIZE` | Maximum size of the advancing datasource connection pool. Worker threads open their own connections, configured separately via `ADVANCING_PICKINGTHREADS` and `ADVANCING_PROCESSINGTHREADS` | `20` | #### Configuring the Advancing controller | Environment variable | Description | Default value | | ---------------------------------------------- | -------------------------------------------------------------------------------- | ---------------------------------------------- | | `ADVANCING_TYPE` | Type of advancing mechanism | `PARALLEL` (alternatives: `KAFKA`, `PARALLEL`) | | `ADVANCING_PICKINGTHREADS` | Number of worker threads for reading from database (picking operations) | `1` | | `ADVANCING_PROCESSINGTHREADS` | Number of threads for parallel processing of advancing events | `20` | | `ADVANCING_PROCESSINGBUFFERSIZE` | Maximum buffer size for processing queue. Controls how many events can be queued | `20` | | `ADVANCING_BLOCKPICKINGIFNOWORKERAVAILABLE` | Block picking operations when no worker threads are available | `true` | | `ADVANCING_PICKINGPAUSEMILLIS` | Pause duration between picking batches (ms) | `50` | | `ADVANCING_COOLDOWNAFTERSECONDS` | Cooldown period after processing a batch (seconds) | `120` | | `ADVANCING_SCHEDULER_HEARTBEAT_CRONEXPRESSION` | Cron expression for the heartbeat | `"*/2 * * * * ?"` | **How the new advancing controller works:** * **Picking threads** (`ADVANCING_PICKINGTHREADS`): Controls how many worker threads read events from the database. This handles only the picking/reading operations. * **Processing buffer** (`ADVANCING_PROCESSINGBUFFERSIZE`): Acts as a queue between picking and processing. When the buffer is full, no new events are read. When there's available space (even just 1 position), that amount of events will be read. * **Processing threads** (`ADVANCING_PROCESSINGTHREADS`): Controls how many threads process the advancing events in parallel. Events are processed instantly if processing threads are available. If all processing threads are busy, events accumulate in the buffer until it reaches capacity. * **Blocking behavior** (`ADVANCING_BLOCKPICKINGIFNOWORKERAVAILABLE`): When enabled, prevents picking operations if no worker threads are available, ensuring better resource management. ### CAS lib configuration | Environment Variable | Description | Default Value | | --------------------- | ---------------------------- | ------------- | | `FLOWX_SPICEDB_HOST` | SpiceDB server hostname | `spicedb` | | `FLOWX_SPICEDB_PORT` | SpiceDB server port | `50051` | | `FLOWX_SPICEDB_TOKEN` | SpiceDB authentication token | - | ### Configuring cleanup mechanism | Environment variable | Description | Default value | | ----------------------------------------- | ------------------------------------------------- | ---------------------------------------------------------- | | `SCHEDULER_THREADS` | Number of threads for the scheduler | `10` | | `SCHEDULER_PROCESSCLEANUP_ENABLED` | Activates the cron job for process cleanup | `false` | | `SCHEDULER_PROCESSCLEANUP_CRONEXPRESSION` | Cron expression for the process cleanup scheduler | `0 */5 0-5 * * ?` (every 5 minutes between 12 AM and 5 AM) | | `SCHEDULER_PROCESSCLEANUP_BATCHSIZE` | Number of processes to clean up in one batch | `1000` | ### Managing subprocesses expiration | Environment variable | Description | Default value | | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | | `FLOWX_PROCESS_EXPIRESUBPROCESSES` | When true, terminates all subprocesses upon parent process expiration. When false, subprocesses follow their individual expiration settings | `true` | ### Configuring application management The following configuration from versions before 4.1 will be deprecated in version 5.0: * `MANAGEMENT_METRICS_EXPORT_PROMETHEUS_ENABLED`: Enables or disables Prometheus metrics export. Starting from version 4.1, use the following configuration instead. This setup is backwards compatible until version 5.0. | Environment variable | Description | Default value | | ---------------------------------------------- | --------------------------------- | ------------- | | `MANAGEMENT_PROMETHEUS_METRICS_EXPORT_ENABLED` | Enables Prometheus metrics export | `false` | *** ## RBAC configuration Process Engine requires specific RBAC permissions for proper access to Kubernetes resources: ```yaml theme={"system"} rbac: create: true rules: - apiGroups: - "" resources: - secrets - configmaps - pods verbs: - get - list - watch ``` *** ## Ingress and CORS The Process Engine is exposed on both the admin and public hosts, with a dedicated route for runtime process instances on the admin host. Routing is configured through the FlowX Helm chart, which renders either a Kubernetes Ingress (default) or a Gateway API HTTPRoute per service. CORS handling lives in the service code; only the allowed-origins list is deployment-specific. ### Service routes | Host group | External path | Backend receives | Notes | | --------------- | ----------------- | -------------------- | ------------------- | | admin | `/onboarding/api` | `/api/...` | Strip `/onboarding` | | public | `/onboarding/api` | `/api/...` | Strip `/onboarding` | | admin-instances | `/api/instances` | `/api/instances/...` | Path preserved | Paths are set through `services.process-engine.ingress..path` (or `gateway..paths`) in the chart values. The admin and public routes share the same backend route family — the `/onboarding` prefix is stripped before forwarding so the backend receives `/api/...`. ### CORS configuration | Environment Variable | Description | Default Value | | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | | `APPLICATION_CORS_ALLOW_ORIGIN` | Comma-separated list of origins allowed to call this service from the browser. Supports wildcard subdomains. Must include every Designer, runtime renderer, and integration domain that calls Process Engine. | `-` | Allowed methods, allowed headers (including `Authorization`, `Content-Type`, `Fx-Workspace-Id`), and credential handling are baked into the service's `application.yaml` with safe defaults. Override these only if you have a non-standard requirement. For the complete route reference, Gateway API HTTPRoute configuration, and route customization, see the [ingress configuration guide](../ingress-configuration). For SSE (Server-Sent Events) communication configuration, refer to the [Events Gateway setup](../events-gateway-setup) guide. *** ### Script engine configuration The process engine uses a native script engine for executing JavaScript and Python business rules. The native engine runs scripts in separate Node.js and Python worker processes. | Environment variable | Description | Default value | | ---------------------------------------------------------------------- | ---------------------------------------------- | ---------------- | | `APPLICATION_SCRIPT_ENGINE_PROVIDER` | Script engine provider (`native` or `graalvm`) | `native` | | `APPLICATION_SCRIPT_ENGINE_NATIVEENGINE_JS_POOLSIZE` | Number of Node.js worker processes | `16` | | `APPLICATION_SCRIPT_ENGINE_NATIVEENGINE_JS_EXECUTIONTIMEOUTMS` | Execution timeout per script (ms) | `5000` | | `APPLICATION_SCRIPT_ENGINE_NATIVEENGINE_JS_MAXEXECUTIONSPERWORKER` | Max executions before a worker is recycled | `10000` | | `APPLICATION_SCRIPT_ENGINE_NATIVEENGINE_JS_MAXPAYLOADSIZEBYTES` | Max input payload size (bytes) | `1048576` (1 MB) | | `APPLICATION_SCRIPT_ENGINE_NATIVEENGINE_PYTHON_POOLSIZE` | Number of Python worker processes | `8` | | `APPLICATION_SCRIPT_ENGINE_NATIVEENGINE_PYTHON_EXECUTIONTIMEOUTMS` | Execution timeout per script (ms) | `10000` | | `APPLICATION_SCRIPT_ENGINE_NATIVEENGINE_PYTHON_MAXEXECUTIONSPERWORKER` | Max executions before a worker is recycled | `10000` | | `APPLICATION_SCRIPT_ENGINE_NATIVEENGINE_PYTHON_MAXPAYLOADSIZEBYTES` | Max input payload size (bytes) | `1048576` (1 MB) | The `native` script engine is the default starting with 5.9.0, replacing the previous GraalVM-based engine. Scripts run in isolated subprocess pools. To revert to GraalVM, set `APPLICATION_SCRIPT_ENGINE_PROVIDER=graalvm`. ### Stuck token recovery Automatically recovers process tokens that get stuck due to transient failures. | Environment variable | Description | Default value | | ----------------------------------------------- | ------------------------------------------------------ | --------------- | | `SCHEDULER_STUCKTOKENRECOVERY_ENABLED` | Turn on or off stuck token recovery | `false` | | `SCHEDULER_STUCKTOKENRECOVERY_CRONEXPRESSION` | Cron expression for recovery sweeps | `0 */5 * * * ?` | | `SCHEDULER_STUCKTOKENRECOVERY_THRESHOLDMINUTES` | Minutes a token must be stuck before recovery triggers | `10` | | `SCHEDULER_STUCKTOKENRECOVERY_BATCHSIZE` | Max tokens recovered per sweep | `100` | | `SCHEDULER_STUCKTOKENRECOVERY_MAXAGEHOURS` | Max age (hours) of stuck tokens to consider | `4` | *** ## Troubleshooting ### Common issues **Symptoms:** The process-engine pod crashes or restarts repeatedly during startup. **Solutions:** 1. Verify PostgreSQL connectivity and that the database exists and is accessible 2. Check Redis connection settings (`SPRING_REDIS_HOST`, `SPRING_REDIS_PORT`) and ensure Redis is running 3. Confirm Kafka bootstrap servers are reachable (`SPRING_KAFKA_BOOTSTRAPSERVERS`) 4. Review pod logs for specific connection errors — the first failing dependency is usually the root cause 5. Ensure all required secrets (database passwords, OAuth credentials) are correctly mounted **Symptoms:** Processes get stuck at certain nodes and do not move forward. **Solutions:** 1. Verify the advancing controller is running and properly connected (`ADVANCING_DATASOURCE_JDBC_URL`) 2. Check that advancing-related Kafka topics exist and are correctly configured (`KAFKA_TOPIC_PROCESS_NOTIFY_ADVANCE`) 3. Review the advancing controller type (`ADVANCING_TYPE`) and thread configuration 4. Ensure the scheduler extension topics are properly set up for timer-based advances 5. Check Kafka consumer group lag for `advance` and `scheduler-advancing` groups **Symptoms:** Process instances are not appearing in search results or the indexing service reports errors. **Solutions:** 1. Confirm `FLOWX_INDEXING_ENABLED` is set to `true` 2. Verify Elasticsearch connection settings (`SPRING_ELASTICSEARCH_REST_URIS`, credentials) 3. Check that the indexing type matches your setup (`FLOWX_INDEXING_TYPE`: `kafka` or `http`) 4. If using Kafka indexing, ensure the `KAFKA_TOPIC_PROCESS_INDEX_OUT` topic exists 5. Verify the Elasticsearch index name and shard configuration are valid **Symptoms:** The process-engine pod experiences OOM kills or high memory consumption. **Solutions:** 1. Review Redis caching configuration — large process definitions and compiled scripts are cached in Redis 2. Check `ADVANCING_PROCESSINGBUFFERSIZE` and `ADVANCING_PROCESSINGTHREADS` — high values increase memory usage 3. Ensure the cleanup mechanism is enabled (`SCHEDULER_PROCESSCLEANUP_ENABLED`) to remove completed process instances 4. Review Kafka consumer thread counts — each thread consumes memory for message buffering 5. Monitor the config params cache settings (`FLOWX_CACHE_CONFIGPARAMS_MAXENTRIES`) and reduce if needed *** ## Related resources Cache and session configuration including Sentinel and Cluster modes Configure process instance indexing and search capabilities Archive and manage historical process instance data Configure role-based access control for process definitions