FlowX Engine setup

​

Infrastructure prerequisites

Before initiating the FlowX.AI Engine, ensure the following infrastructure components are properly installed and configured:

• Kafka: Version 2.8 or higher.
• Elasticsearch: Version 7.11.0 or higher.
• PostgreSQL - version 13 or higher for storing application data (could vary based on your preferred relational database)
• MongoDB - version 4.4 or higher for managing runtime builds

​

Dependencies

The FlowX Engine interacts with various components critical for its operation:

• Database: Primary storage for the engine.
• Redis Server: Used for caching purposes. Refer to Redis Configuration.
• Kafka: Facilitates messaging and event-driven architecture. Details on Configuring Kafka.

For a microservices architecture, it’s common for services to manage their data via dedicated databases.

​

Required External Services

• Redis Cluster: Essential for caching process definitions, compiled scripts, and Kafka responses.
• Kafka Cluster: Serves as the communication backbone with external plugins and integration.

​

Configuration Setup

FlowX.AI Engine utilizes environment variables for configuration. Below are the key environment variables you need to configure:

• Database configuration
• Configuring authorization and access roles
• Configuring the data source
• Configuring Redis
• Configuring logging
• Configuring Kafka
• Configuring access roles for processes

​

Database configuration

​

PostgreSQL

• SPRING_DATASOURCE_URL - Database URL for PostgreSQL
• SPRING_DATASOURCE_USERNAME - Username for PostgreSQL
• SPRING_DATASOURCE_PASSWORD - Password for PostgreSQL
• SPRING_DATASOURCE_DRIVER_CLASS_NAME - Driver class for PostgreSQL

​

Configuring MongoDB (runtime database - additional data)

The Process Engine requires to connect to Runtime MongoDB instance to access runtime build information. Use the following environment variables for configuration:

• SPRING_DATA_MONGODB_RUNTIME_ENABLED - Enables runtime MongoDB usage
  • Default Value: true
• SPRING_DATA_MONGODB_RUNTIME_URI - URI for connecting to MongoDB for Runtime DB (app-runtime)
  • Format: mongodb://${DB_USERNAME}:${DB_PASSWORD}@<host1>,<host2>,<arbiter-host>:<port>/<database>?retryWrites=false
• DB_USERNAME: app-runtime

​

Authorization & access roles

This section outlines the OAuth2 configuration settings for securing the Spring application, including resource server settings, security type, and access authorizations.

​

Resource server settings (OAuth2 configuration)

​

Service account settings

This section contains the environment variables for configuring process engine service account.

More details about the necessary service account can be found here.

​

Security configuration

​

Configuring Kafka

Kafka handles all communication between the FlowX.AI Engine and external plugins and integrations. It is also used for notifying running process instances when certain events occur.

​

Kafka connection settings

​

Kafka consumer retry settings

​

Consumer groups & consumer threads configuration

Both a producer and a consumer must be configured:

​

Consumer groups & consumer threads

In Kafka a consumer group is a group of consumers that jointly consume and process messages from one or more Kafka topics. Each consumer group has a unique identifier called a group ID, which is used by Kafka to manage message consumption and distribution among the members of the group.

Thread numbers, on the other hand, refer to the number of threads that a consumer application uses to process messages from Kafka. By default, each consumer instance runs in a single thread, which can limit the throughput of message processing. Increasing the number of consumer threads can help to improve the parallelism and efficiency of message consumption, especially when dealing with high message volumes.

Both group IDs and thread numbers can be configured in Kafka to optimize the processing of messages according to specific requirements, such as message volume, message type, and processing latency.

The configuration related to consumers (group ids and thread numbers) can be configured separately for each message type as it follows:

​

Consumer group configuration

​

Consumer thread configuration

It is important to know that all the events that start with a configured pattern will be consumed by the Engine. This makes it possible to create a new integration and connect it to the engine without changing the configuration.

​

Configuring Kafka topics

 topic:
    naming:
      package: "ai.flowx."
      environment: "dev."
      version: ".v1"
      prefix: ${kafka.topic.naming.package}${kafka.topic.naming.environment}
      suffix: ${kafka.topic.naming.version}
      engineReceivePattern: engine.receive.

    pattern: ${kafka.topic.naming.prefix}${kafka.topic.naming.engineReceivePattern}*

​

Topics related to the Task Management plugin

​

OPERATIONS_IN request example

{
  "operationType": "UNASSIGN", //type of operation performed in Task Management plugin
  "taskId": "some task id",
  "processInstanceUuid": "1cff0b7d-966b-4b35-9e9b-63b1d6757ec6",
  "swimlaneName": "Default",
  "swimlaneId": "51ec1241-fe06-4576-9c84-31598c05c527",
  "owner": {
    "firstName": null,
    "lastName": null,
    "username": "service-account-flowx-process-engine-account",
    "enabled": false
  },
  "author": "admin@flowx.ai"
}

​

BULK_IN request example

{

  "operations": [
    {
      "operationType": "HOLD",
      "taskId": "some task id",
      "processInstanceUuid": "d3aabfd8-d041-4c62-892f-22d17923b223", // the id of the process instance
      "swimlaneName": "Default", //name of the swimlane
      "owner": null,
      "author": "john.doe@flowx.ai",
    },

    {
      "operationType": "HOLD",
      "taskId": "some task id",
      "processInstanceUuid": "d3aabfd8-d041-4c62-892f-22d17923b223",
      "swimlaneName": "Default", //name of the swimlane
      "owner": null,
      "author": "john.doe@flowx.ai",
    }
  ]
}      

{"processInstanceId": ${processInstanceId}, "callbackTopic": "test.operations.out", "requestID":"1234567890"}

​

Topics related to the scheduler extension

Scheduler

​

Topics related to Timer Events

​

Topics related to the Search Data service

​

Topics related to the Audit service

​

Topics related to ES indexing

​

Processes that can be started by sending messages to a Kafka topic

​

Topics related to Message Events

​

Topics related to Events-gateway microservice

​

Configuring file upload size

The maximum file size allowed for uploads can be set by using the following environment variables:

​

Connecting the Advancing controller

To use advancing controller, the following env vars are needed for process-engine to connect to Advancing Postgres DB:

​

Configuring the Advancing controller

​

Configuring cleanup mechanism

This section contains environment variables that configure the scheduler’s behavior, including thread count, cron jobs for data partitioning, process cleanup, and master election.

​

Managing subprocesses expiration

This section details the environment variable that controls the expiration of subprocesses within a parent process. It determines whether subprocesses should terminate when the parent process expires or follow their own expiration settings.

​

Configuring application management

The FlowX helm chart provides a management service with the necessary parameters to integrate with the Prometheus operator. However, this integration is disabled by default.

To configure Prometheus metrics export for the FlowX.AI Engine, the following environment variable is required:

​

RBAC configuration

Process Engine requires specific RBAC (Role-Based Access Control) permissions to ensure proper access to Kubernetes resources like pods for database lock management. This configuration enables the necessary RBAC rules.

• rbac.create: Set to false to avoid creating additional RBAC resources by default. Custom rules can still be added if required.
• rbac.rules: Define custom RBAC rules for database locking, as shown below.

rbac:
  create: true
  rules:
    - apiGroups:
        - ""
      resources:
        - secrets
        - configmaps
        - pods
      verbs:
        - get
        - list
        - watch