​

Infrastructure prerequisites

The FlowX Engine requires the following components to be set up before it can be started:

• Docker Engine (version 17.06 or higher)
• Kafka (version 2.8 or higher)
• Elasticsearch (version 7.11.0 or higher)
• Database instance

​

Dependencies

The FlowX Engine has dependencies on the following components:

• Database
• Redis server
• Kafka

For Microservices architecture, some Microservices holds their data individually using separate Databases.

A basic Postgres configuration can be set up using a helm values.yaml file as it follows:

• helm values.yaml:

    onboardingdb:
      existingSecret: {{secretName}}
      metrics:
        enabled: true
        service:
          annotations:
            prometheus.io/port: {{prometheus port}}
            prometheus.io/scrape: "true"
          type: ClusterIP
        serviceMonitor:
          additionalLabels:
            release: prometheus-operator
          enabled: true
          interval: 30s
          scrapeTimeout: 10s
      persistence:
        enabled: true
        size: 1Gi
      postgresqlDatabase: onboarding
      postgresqlExtendedConf:
        maxConnections: 200
        sharedBuffers: 128MB
      postgresqlUsername: postgres
      resources:
        limits:
          cpu: 6000m
          memory: 2048Mi
        requests:
          cpu: 200m
          memory: 512Mi
  

​

Redis server

A Redis cluster is required for caching process definitions, compiled scripts, and Kafka responses.

​

Kafka cluster

Kafka serves as the backbone of the engine, facilitating communication with external plugins and integrations.

​

Configuration

FlowX.AI Engine utilizes environment variables for configuration. Below are the key environment variables you need to configure:

• Configuring authorization and access roles
• Configuring the data source
• Configuring Redis
• Configuring logging
• Configuring Kafka
• Configuring access roles for processes

​

Authorization & access roles

This section outlines the OAuth2 configuration settings for securing the Spring application, including resource server settings, security type, and access authorizations.

​

Resource Server Settings (OAuth2 Configuration)

​

Service account settings

This section contains the environment variables for configuring process engine service account.

More details about the necessary service account, here:

Process engine service account

​

Security configuration

​

Configuring Kafka

Kafka handles all communication between the FlowX.AI Engine and external plugins and integrations. It is also used for notifying running process instances when certain events occur.

​

Kafka connection settings

​

Kafka consumer retry settings

​

Consumer groups & consumer threads configuration

Both a producer and a consumer must be configured:

​

Consumer groups & consumer threads

In Kafka a consumer group is a group of consumers that jointly consume and process messages from one or more Kafka topics. Each consumer group has a unique identifier called a group ID, which is used by Kafka to manage message consumption and distribution among the members of the group.

Thread numbers, on the other hand, refer to the number of threads that a consumer application uses to process messages from Kafka. By default, each consumer instance runs in a single thread, which can limit the throughput of message processing. Increasing the number of consumer threads can help to improve the parallelism and efficiency of message consumption, especially when dealing with high message volumes.

Both group IDs and thread numbers can be configured in Kafka to optimize the processing of messages according to specific requirements, such as message volume, message type, and processing latency.

The configuration related to consumers (group ids and thread numbers) can be configured separately for each message type as it follows:

​

Consumer Group Configuration

​

Consumer Thread Configuration

It is important to know that all the events that start with a configured pattern will be consumed by the engine. This makes it possible to create a new integration and connect it to the engine without changing the configuration of the engine.

​

Configuring kafka topics

 topic:
    naming:
      package: "ai.flowx."
      environment: "dev."
      version: ".v1"
      prefix: ${kafka.topic.naming.package}${kafka.topic.naming.environment}
      suffix: ${kafka.topic.naming.version}
      engineReceivePattern: engine.receive.

    pattern: ${kafka.topic.naming.prefix}${kafka.topic.naming.engineReceivePattern}*

​

Topics related to the Task Management plugin

​

OPERATIONS_IN request example

{
  "operationType": "UNASSIGN", //type of operation performed in Task Management plugin
  "taskId": "some task id",
  "processInstanceUuid": "1cff0b7d-966b-4b35-9e9b-63b1d6757ec6",
  "swimlaneName": "Default",
  "swimlaneId": "51ec1241-fe06-4576-9c84-31598c05c527",
  "owner": {
    "firstName": null,
    "lastName": null,
    "username": "service-account-flowx-process-engine-account",
    "enabled": false
  },
  "author": "admin@flowx.ai"
}

​

BULK_IN request example

{

  "operations": [
    {
      "operationType": "HOLD",
      "taskId": "some task id",
      "processInstanceUuid": "d3aabfd8-d041-4c62-892f-22d17923b223", // the id of the process instance
      "swimlaneName": "Default", //name of the swimlane
      "owner": null,
      "author": "john.doe@flowx.ai",
    },

    {
      "operationType": "HOLD",
      "taskId": "some task id",
      "processInstanceUuid": "d3aabfd8-d041-4c62-892f-22d17923b223",
      "swimlaneName": "Default", //name of the swimlane
      "owner": null,
      "author": "john.doe@flowx.ai",
    }
  ]
}      

{"processInstanceId": ${processInstanceId}, "callbackTopic": "test.operations.out", "requestID":"1234567890"}

​

Topics related to the scheduler extension

Scheduler

​

Topics related to Timer Events

​

Topics related to the Search Data service

​

Topics related to the Audit service

​

Topics related to ES indexing

​

Processes that can be started by sending messages to a Kafka topic

​

Topics related to Message Events

​

Topics related to events-gateway microservice

​

Configuring file upload size

The maximum file size allowed for uploads can be set by using the following environment variables:

​

Configuring Advancing controller

To use advancing controller, the following env vars are needed for process-engine to connect to Advancing Postgres DB:

​

Configuring cleanup mechanism

This section contains environment variables that configure the scheduler’s behavior, including thread count, cron jobs for data partitioning, process cleanup, and master election.

​

Managing Subprocesses Expiration

This section details the environment variable that controls the expiration of subprocesses within a parent process. It determines whether subprocesses should terminate when the parent process expires or follow their own expiration settings.

​

Partitioning and archiving

For more details about process instance partitioning and archiving, including configuration of the feature, see: