Integration designer setup

​

Infrastructure prerequisites

The Integration Designer service requires the following components to be set up before it can be started:

• PostgreSQL - version 13 or higher for managing advancing data source
• MongoDB - version 4.4 or higher for managing integration and runtime data
• Kafka - version 2.8 or higher for event-driven communication between services
• OAuth2 Authentication - Ensure a Keycloak server or compatible OAuth2 authorization server is configured

​

Dependencies

• Database configuration
• Kafka configuration
• Authentication & access roles
• Logging

​

Configuration

​

Config profile

• CONFIG_PROFILE - This environment variable must be set explicitly and exactly to ensure that no unintended profiles are loaded by mistake. The value of this variable should represent a minimal configuration state, relying only on defaults specified in the application.properties file of the application.

Example:

- name: CONFIG_PROFILE
  value: k8stemplate_v2,kafka-auth

​

Database configuration

Integration Designer uses both PostgreSQL and MongoDB for managing advancing data and integration information. Configure these database connections with the following environment variables:

​

PostgreSQL (Advancing data source)

• ADVANCING_DATASOURCE_URL - Database URL for the advancing data source in PostgreSQL
• ADVANCING_DATASOURCE_USERNAME - Username for the advancing data source in PostgreSQL
• ADVANCING_DATASOURCE_PASSWORD - The password for the advancing data source in PostgreSQL.
  • Details: Must match the credentials configured in the PostgreSQL database.
• ADVANCING_DATASOURCE_DRIVER_CLASS_NAME - The database driver class name for the advancing data source
  • Details: Required to ensure proper database connectivity. This value can be overridden for other databases, such as Oracle.
    • For PostgreSQL: org.postgresql.Driver
    • For Oracle: oracle.jdbc.OracleDriver

​

MongoDB (Integration data and runtime data)

Integration Designer requires two MongoDB databases for managing integration-specific data and runtime data. The integration-designer database is dedicated to Integration Designer, while the shared app-runtime database supports multiple services.

• Integration Designer Database (integration-designer): Stores data specific to Integration Designer, such as integration configurations, metadata, and other operational data.
• Shared Runtime Database (app-runtime): Shared across multiple services, this database manages runtime data essential for integration and data flow execution.

To connect these MongoDB databases, configure the following environment variables:

• SPRING_DATA_MONGODB_URI- URI for connecting to the Integration Designer MongoDB instance
  • Format: mongodb://${DB_USERNAME}:${DB_PASSWORD}@<host1>,<host2>,<arbiter-host>:<port>/<database>?retryWrites=false
• DB_USERNAME: integration-designer
• SPRING_DATA_MONGODB_STORAGE - Specifies the storage type used for the Runtime MongoDB instance (Azure environments only)
  • Possible Values: mongodb, cosmosdb
  • Default Value: mongodb

Enable Runtime MongoDB:

• SPRING_DATA_MONGODB_RUNTIME_URI - URI for connecting to MongoDB for Runtime MongoDB (app-runtime)
  • Format: SPRING_DATA_MONGODB_RUNTIME_URI : mongodb://${DB_USERNAME}:${DB_PASSWORD}@<host1>,<host2>,<arbiter-host>:<port>/<database>??retryWrites=false
• DB_USERNAME: app-runtime

​

Configuring Kafka

To configure Kafka for Integration Designer, set the following environment variables. This configuration includes naming patterns, consumer group settings, and retry intervals for authentication exceptions.

​

General Kafka configuration

• SPRING_KAFKA_BOOTSTRAP_SERVERS - Address of the Kafka server in the format host:port
• KAFKA_TOPIC_NAMING_ENVIRONMENT - Environment-specific suffix for Kafka topics
• FLOWX_WORKFLOW_CREATETOPICS - To automatically create kafka topics for development environments
  • When set to true: In development environments, where Kafka topics may need to be created automatically, this configuration can be enabled (flowx.workflow.createTopics: true). This allows for the automatic creation of “in” and “out” topics when workflows are created, eliminating the need to wait for topic creation at runtime.
  • Default setting (false): In production or controlled environments, where automated topic creation is not desired, this setting remains false to prevent unintended Kafka topic creation.

​

Kafka consumer settings

• KAFKA_CONSUMER_GROUP_ID_START_WORKFLOWS - Consumer group ID for starting workflows

  • Default Value: start-workflows-group

• KAFKA_CONSUMER_THREADS_START_WORKFLOWS - Number of Kafka consumer threads for starting workflows

  • Default Value: 3

• KAFKA_AUTH_EXCEPTION_RETRY_INTERVAL - Interval (in seconds) between retries after an AuthorizationException

  • Default Value: 10

​

Kafka topic naming structure

The Kafka topics for Integration Designer use a structured naming convention with dynamic components, allowing for easy integration across environments. This setup defines separators, environment identifiers, and specific naming patterns for both engine and integration-related messages.

​

Topic naming components

Predefined patterns for services

• Engine Receive Pattern - kafka.topic.naming.engineReceivePattern

  • Pattern: engine${dot}receive${dot}
  • Example Topic Prefix: ai.flowx.dev.engine.receive.

• Integration Receive Pattern - kafka.topic.naming.integrationReceivePattern

  • Pattern: integration${dot}receive${dot}
  • Example Topic Prefix: ai.flowx.dev.integration.receive.

​

Kafka topics

• Events Gateway - Outgoing Messages

  • Topic: ${kafka.topic.naming.prefix}eventsgateway${dot}receive${dot}workflowinstances${kafka.topic.naming.suffix}
  • Purpose: Topic for outgoing workflow instance messages from the events gateway
  • Example Value: ai.flowx.dev.eventsgateway.receive.workflowinstances.v1

• Engine Pattern

  • Pattern: ${kafka.topic.naming.prefix}${kafka.topic.naming.engineReceivePattern}
  • Purpose: Topic pattern for receiving messages by the engine service
  • Example Value: ai.flowx.dev.engine.receive.*

• Integration Pattern

  • Pattern: ${kafka.topic.naming.prefix}${kafka.topic.naming.integrationReceivePattern}*
  • Purpose: Topic pattern for receiving messages by the integration service
  • Example Value: ai.flowx.dev.integration.receive.*

​

Configuring authentication and access roles

Integration Designer uses OAuth2 for secure access control. Set up OAuth2 configurations with these environment variables:

• SECURITY_OAUTH2_BASE_SERVER_URL - Base URL for the OAuth 2.0 Authorization Server
• SECURITY_OAUTH2_CLIENT_CLIENT_ID - Unique identifier for the client application registered with the OAuth 2.0 server
• SECURITY_OAUTH2_CLIENT_CLIENT_SECRET - Secret key for authenticating requests made by the authorization client
• SECURITY_OAUTH2_REALM - The realm name for OAuth2 authentication
• SECURITY_OAUTH2_SERVICE_ACCOUNT_ADMIN_CLIENT_ID - Client ID for the integration designer service account
• SECURITY_OAUTH2_SERVICE_ACCOUNT_ADMIN_CLIENT_SECRET - Client Secret for the integration designer service account

Refer to the dedicated section for configuring user roles and access rights:

Refer to the dedicated section for configuring a service account:

​

Authentication and access roles

• SECURITY_OAUTH2_BASE_SERVER_URL - Base URL for the OAuth2 authorization server
• SECURITY_OAUTH2_REALM - Realm for OAuth2 authentication
• SECURITY_OAUTH2_CLIENT_CLIENT_ID - Client ID for the Integration Designer OAuth2 client
• SECURITY_OAUTH2_CLIENT_CLIENT_SECRET - Client Secret for the Integration Designer OAuth2 client
• SECURITY_OAUTH2_SERVICE_ACCOUNT_ADMIN_CLIENT_ID - Client ID for the Keycloak admin service account
• SECURITY_OAUTH2_SERVICE_ACCOUNT_ADMIN_CLIENT_SECRET - Client Secret for the Keycloak admin service account

​

Configuring loogging

To control the log levels for Integration Designer, set the following environment variables:

• LOGGING_LEVEL_ROOT - The log level for root Spring Boot microservice logs
• LOGGING_LEVEL_APP - The log level for application-level logs

​

Configuring admin ingress

Integration Designer provides an admin ingress route, which can be enabled and customized with additional annotations for SSL certificates or routing preferences.

• Enabled: Set to true to enable the admin ingress route.
• Hostname: Define the hostname for admin access.

  ingress:
    enabled: true
    admin:
      enabled: true
      hostname: "{{ .Values.flowx.ingress.admin }}"
      path: /integration(/|$)(.*)
      annotations:
        nginx.ingress.kubernetes.io/rewrite-target: /$2
        nginx.ingress.kubernetes.io/cors-allow-headers: DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization,platform,Flowx-Platform

​

Monitoring and maintenance

To monitor the performance and health of the Application Manager, use tools like Prometheus or Grafana. Configure Prometheus metrics with the following environment variable:

• MANAGEMENT_PROMETHEUS_METRICS_EXPORT_ENABLED - Enables or disables Prometheus metrics export (default: false).

​

RBAC configuration

Integration Designer requires specific RBAC (Role-Based Access Control) permissions to access Kubernetes ConfigMaps and Secrets, which store necessary configurations and credentials. Set up these permissions by enabling RBAC and defining the required rules.

• rbac.create: Set to true to create RBAC resources.
• rbac.rules: Define custom RBAC rules as follows:

rbac:
  create: true
  rules:
    - apiGroups:
        - ""
      resources:
        - secrets
        - configmaps
        - pods
      verbs:
        - get
        - list
        - watch

This configuration grants read access (get, list, watch) to ConfigMaps, Secrets, and Pods, which is essential for retrieving application settings and credentials required by Integration Designer.