Application manager setup

The Application manager is microservice that manages applications and also acts as a proxy for front-end requests related to resources.

​

Infrastructure prerequisites

Before you start setting up the Application Manager service, ensure the following infrastructure components are in place:

• PostgreSQL: Version 13 or higher for storing application data (based on your preferred relational database).
• MongoDB: Version 4.4 or higher for managing runtime builds.
• Redis: Version 6.0 or higher for caching needs.
• Kafka: Version 2.8 or higher for messaging and event-driven communication between services.

​

Dependencies

The Application Manager relies on other FlowX services and components to function properly:

• Database instance: For storing application details, manifests, and configurations.
• Authorization & Access Management: For securing access to resources and managing roles.
• Kafka Event Bus: For enabling event-driven operations.
• Proxy Mechanism: To forward resource-related requests to appropriate services.

​

Configuration

​

Environment variables

• APP_MANAGER_DB_URL- Connection string for the relational database
• APP_MANAGER_DB_USER - Username for the database
• APP_MANAGER_DB_PASSWORD - Password for the database
• APP_MANAGER_DB_NAME - Database name

​

Configuring authorization and access roles

To integrate the Application Manager with the identity management system for authorization, set the following environment variables:

• SECURITY_OAUTH2_BASE_SERVER_URL - Base URL for the OAuth 2.0 Authorization Server
• SECURITY_OAUTH2_CLIENT_CLIENT_ID - Unique identifier for the client application registered with the OAuth 2.0 server
• SECURITY_OAUTH2_CLIENT_CLIENT_SECRET - Secret key for authenticating requests made by the authorization client
• SECURITY_OAUTH2_REALM - The realm name for OAuth2 authentication
• SECURITY_OAUTH2_SERVICE_ACCOUNT_ADMIN_CLIENT_ID - Client ID for the application manager service account
• SECURITY_OAUTH2_SERVICE_ACCOUNT_ADMIN_CLIENT_SECRET - Client Secret for the application manager service account

Refer to the dedicated section for configuring user roles and access rights:

​

Database configuration

Configure the data sources for PostgreSQL and MongoDB as follows:

​

PostgreSQL (application data)

• SPRING_DATASOURCE_URL - Database URL for PostgreSQL
• SPRING_DATASOURCE_USERNAME - Username for PostgreSQL
• SPRING_DATASOURCE_PASSWORD - Password for PostgreSQL
• SPRING_DATASOURCE_DRIVER_CLASS_NAME - Driver class for PostgreSQL

​

Configuring MongoDB (runtime database - additional data)

The Application Manager requires MongoDB to store runtime build information. Use the following environment variables for configuration:

• SPRING_DATA_MONGODB_URI - URI for connecting to the MongoDB instance -> to connect to app-runtime database
  • Format: mongodb://${DB_USERNAME}:${DB_PASSWORD}@<host1>,<host2>,<arbiter-host>:<port>/${DB_NAME}?retryWrites=false
• DB_USERNAME : app-runtime
• DB_NAME: app-runtime
• DB_PASSWORD: DB password.
• SPRING_DATA_MONGODB_STORAGE - Specifies the storage type used for the Runtime MongoDB instance (Azure environments only)
  • Possible Values: mongodb, cosmosdb
  • Default Value: mongodb

​

Configuring Redis

If caching is required, configure Redis using the following environment variables:

• SPRING_DATA_REDIS_HOST - Hostname or IP address of the Redis server.
• SPRING_DATA_REDIS_PASSWORD - Password for authenticating with the Redis server.
• SPRING_DATA_REDIS_PORT - Hostname of the Redis server. This specifies where the Redis server is running and should be accessible from the application.

​

Configuring Rest request size

To control the size of REST requests handled by the Application Manager, use the following environment variable:

• FLOWX_RUNTIMEEXECUTIONPROXY_WEBCLIENT_MAXINMEMORYSIZE - Specifies the maximum size (in bytes) of in-memory data for REST requests. This is particularly useful when dealing with large payloads to prevent excessive memory consumption.
  • Default Value: 5242880 (5 MB)
  • Usage Example: Set to 10485760 (10 MB) to allow larger in-memory request sizes.

​

Configuring scheduler

The Application Manager scheduler supports retrying failed deployments and master election:

​

Retry failed deployments

• Configuration:
  • FLOWX_SCHEDULER_RETRY_FAILED_DEPLOYMENTS_CRON: "0 * * * * *"
• Purpose: Configures a cron job to retry updating builds in the runtime database every minute when previous attempts have failed.

​

Master election

• Configuration:
  • FLOWX_SCHEDULER_MASTER_ELECTION_ENABLED: true
  • FLOWX_SCHEDULER_MASTER_ELECTION_CRON_EXPRESSION: "*/30 * * * * *"
  • FLOWX_SCHEDULER_MASTER_ELECTION_PROVIDER: redis
• Purpose: Enables master election for improved scheduling coordination.

​

Configuring Kafka

The Application Manager uses Kafka for event-driven operations. Set up the Kafka configuration using the following environment variables:

• SPRING_KAFKA_BOOTSTRAP_SERVERS - Address of the Kafka server, formatted as “host”
• SPRING_KAFKA_CONSUMER_GROUP_ID - Consumer group ID for Kafka topics
• KAFKA_CONSUMER_THREADS - Number of Kafka consumer threads
• FLOWX_KAFKA_PAYLOAD_SIZE_LIMIT - Maximum payload size for Kafka messages (default: 512000).

​

Kafka consumer configuration

Configure Kafka consumer groups and threads for efficient handling of build-related topics:

• Consumer Groups:

  • Application Resource:
    • KAFKA_CONSUMER_GROUP_ID_APPLICATION_RESOURCE_EXPORT: appResourceExportGroup
    • KAFKA_CONSUMER_GROUP_ID_APPLICATION_RESOURCE_IMPORT: appResourceImportGroup
    • Build Resource:
      • KAFKA_CONSUMER_GROUP_ID_BUILD_CREATE: buildCreateGroup
      • KAFKA_CONSUMER_GROUP_ID_BUILD_UPDATE: buildUpdateGroup
      • KAFKA_CONSUMER_GROUP_ID_BUILD_RESOURCE_EXPORT: buildResourceExportGroup
      • KAFKA_CONSUMER_GROUP_ID_BUILD_RESOURCE_IMPORT: buildResourceImportGroup
    • Process:
      • KAFKA_CONSUMER_GROUP_ID_PROCESS_START: processStartGroup
    • Merge:
      • KAFKA_CONSUMER_GROUP_ID_APPLICATION_MERGE: appMergeItemGroup

• Consumer Threads:

  • Application Resource:
    • KAFKA_CONSUMER_THREADS_APPLICATION_RESOURCE_EXPORT: 3
    • KAFKA_CONSUMER_THREADS_APPLICATION_RESOURCE_IMPORT: 3
  • Build Resource:
    • KAFKA_CONSUMER_THREADS_BUILD_CREATE: 2
    • KAFKA_CONSUMER_THREADS_BUILD_UPDATE: 4
    • KAFKA_CONSUMER_THREADS_BUILD_RESOURCE_EXPORT: 3
    • KAFKA_CONSUMER_THREADS_BUILD_RESOURCE_IMPORT: 3
  • Merge:
    • KAFKA_CONSUMER_THREADS_APPLICATION_MERGE: 3

​

Topic naming components

The Application Manager uses a structured naming convention for Kafka topics, designed to support flexibility through pre-defined components in the topic structure. These components, while not directly configurable, allow optional modifications when the desired version cannot be obtained through $package . $environment . $separator . $version.

Each topic adheres to a consistent naming schema for streamlined communication across environments and versions.

​

Application resource topics

• Resource Export -**Environment variable: KAFKA_TOPIC_APPLICATION_RESOURCE_EXPORT

  • Pattern: ${kafka.topic.naming.prefix}application${separator2}version${separator}export${kafka.topic.naming.suffix}
  • Purpose: For exporting application resources.
  • Example: ai.flowx.application-version.export.v1

• Resource Import

  • Environment variable: KAFKA_TOPIC_APPLICATION_RESOURCE_IMPORT
  • Pattern: ${kafka.topic.naming.prefix}application${separator2}version${separator}import${kafka.topic.naming.suffix}
  • Purpose: For importing application resources.
  • Example: ai.flowx.application-version.import.v1

​

Build resource topics

• Build Export

  • Environment variable: KAFKA_TOPIC_BUILD_RESOURCE_EXPORT
  • Pattern: ${kafka.topic.naming.prefix}build${separator}export${kafka.topic.naming.suffix}
  • Purpose: For exporting build resources.
  • Example: ai.flowx.build.export.v1

• Build Import

  • Environment variable: KAFKA_TOPIC_BUILD_RESOURCE_IMPORT
  • Pattern: ${kafka.topic.naming.prefix}build${separator}import${kafka.topic.naming.suffix}
  • Purpose: For importing build resources.
  • Example: ai.flowx.build.import.v1

• Resource Create:

  • Environment variable: KAFKA_TOPIC_BUILD_RESOURCE_CREATE
  • Pattern: ${kafka.topic.naming.prefix}build${dot}create${kafka.topic.naming.suffix}
  • Purpose: Handles creation of build resources.
  • Example: ai.flowx.build.create.v1

• Resource Update:

  • Environment variable: KAFKA_TOPIC_BUILD_RESOURCE_UPDATE
  • Pattern: ${kafka.topic.naming.prefix}build${dot}update${kafka.topic.naming.suffix}
  • Purpose: Handles updating of build resources.
  • Example: ai.flowx.build.update.v1

​

Start timer events

• Timer Events Updates:
  • Environment variable: KAFKA_TOPIC_START_TIMER_EVENTS_UPDATES
  • Pattern: ${kafka.topic.naming.prefix}build${dot}start${dash}timer${dash}events${dot}updates${dot}in${kafka.topic.naming.suffix}
  • Purpose: Sends updates from the admin to the application manager about start timer events, specifically if they have been edited or deleted.
  • Example: ai.flowx.build.start-timer-events.updates.in.v1

​

Process topics

• Start for Event

  • Environment variable: KAFKA_TOPIC_PROCESS_START_FOR_EVENT
  • Pattern: ${kafka.topic.naming.prefix}core${separator}trigger${separator}start${separator2}for${separator2}event${separator}process${kafka.topic.naming.suffix}
  • Purpose: For triggering process start events.
  • Example: ai.flowx.core.trigger.start-for-event.process.v1

• Start by Name

  • Environment variable: KAFKA_TOPIC_PROCESS_START_BY_NAME
  • Pattern: ${kafka.topic.naming.prefix}core${separator}trigger${separator}start${separator2}by${separator2}name${separator}process${kafka.topic.naming.suffix}
  • Purpose: For triggering process start by name events.
  • Example: ai.flowx.core.trigger.start-by-name.process.v1
    • Output Topic:
    • Environment variable: KAFKA_TOPIC_PROCESS_START_BY_NAME_OUT
    • Pattern: ${kafka.topic.naming.prefix}core${separator}trigger${separator}start${separator2}by${separator2}name${separator}process${separator}out${kafka.topic.naming.suffix}
    • Example: ai.flowx.core.trigger.start-by-name.process.out.v1

• Scheduled Timer Events

  • Set Timer Schedule
    • Environment variable: KAFKA_TOPIC_PROCESS_SCHEDULED_TIMER_EVENTS_SET
    • Pattern: ${kafka.topic.naming.prefix}core${separator}trigger${separator}set${separator}timer${separator2}event${separator2}schedule${kafka.topic.naming.suffix}
    • Purpose: For setting scheduled timer events.
    • Example: ai.flowx.core.trigger.set.timer-event-schedule.v1
  • Stop Timer Schedule
    • Environment variable: KAFKA_TOPIC_PROCESS_SCHEDULED_TIMER_EVENTS_STOP
    • Pattern: ${kafka.topic.naming.prefix}core${separator}trigger${separator}stop${separator}timer${separator2}event${separator2}schedule${kafka.topic.naming.suffix}
    • Purpose: For stopping scheduled timer events.
    • Example: ai.flowx.core.trigger.stop.timer-event-schedule.v1

​

Audit topics

• Audit Output
  • Environment variable: KAFKA_TOPIC_AUDIT_OUTPUT
  • Pattern: ${kafka.topic.naming.prefix}core${separator}trigger${separator}save${separator}audit${kafka.topic.naming.suffix}
  • Purpose: For sending audit-related events.
  • Example: ai.flowx.core.trigger.save.audit.v1

​

Merge topics

• Environment variable: KAFKA_TOPIC_MERGE
• Pattern: ${kafka.topic.naming.prefix}application${dash}version${dot}merge${kafka.topic.naming.suffix}
• Purpose: For sending merge-related messages, including commands to initiate resource merges and notifications for the completion of individual resource merges or the overall merge process. The merge is performed in parallel across all resources.
• Example: ai.flowx.application-version.merge.v1

​

Configuring resource proxy

The Resource Proxy module forwards resource-related requests to appropriate services, handling CRUD operations on the manifest. It requires proper configuration of proxy endpoints:

• RESOURCE_PROXY_MANIFEST_URL - URL for managing the application manifest
• RESOURCE_PROXY_TARGET_URL - URL for forwarding resource-related requests to their respective services

​

Configuring logging

To control the logging levels for the Application Manager, use the following environment variables:

• LOGGING_LEVEL_ROOT - Log level for the root service logs
• LOGGING_LEVEL_APP - Log level for application-level logs
• LOGGING_LEVEL_DB - Log level for database interactions

​

Configuring file storage

If the Application Manager requires file storage for resources or builds, configure S3-compatible storage using the following environment variables:

• APPLICATION_FILE_STORAGE_TYPE: Type of storage to use. Set this to s3 to enable S3-compatible file storage.
• APPLICATION_FILE_STORAGE_DELETION_STRATEGY: Strategy for handling file deletion. Options include:
  • delete: Files are permanently deleted. (DEFAULT)
  • disabled: File deletion is disabled.
  • deleteBypassingGovernanceRetention: Files are deleted, bypassing governance retention settings.
• APPLICATION_FILE_STORAGE_S3_ENABLED: Set to true to enable S3-compatible storage.
• APPLICATION_FILE_STORAGE_S3_SERVER_URL: URL of the S3-compatible storage server.
• APPLICATION_FILE_STORAGE_S3_ENCRYPTION_ENABLED: Set to true to enable server-side encryption for stored files. Default: false.
• APPLICATION_FILE_STORAGE_S3_ACCESS_KEY: Access key for the S3 storage.
• APPLICATION_FILE_STORAGE_S3_SECRET_KEY: Secret key for the S3 storage.
• APPLICATION_FILE_STORAGE_S3_BUCKET_PREFIX: Prefix for S3 bucket names. Default: "applications-bucket".

​

Data model overview

The Application Manager stores application data using a relational database schema, with key entities such as application, application_version, and application_manifest. Below are descriptions of primary entities:

• Application - Defines an application with its details like name, type, and metadata.
• Application Branch - Represents branches for versioning within an application.
• Application Version - Keeps track of each version of an application, including committed and WIP statuses.
• Application Manifest - Contains the list of resources associated with a specific application version.

​

Monitoring and maintenance

To monitor the performance and health of the Application Manager, use tools like Prometheus or Grafana. Configure Prometheus metrics with the following environment variable:

• MANAGEMENT_PROMETHEUS_METRICS_EXPORT_ENABLED - Enables or disables Prometheus metrics export (default: false).

​

Ingress configuration

Configure ingress to control external access to Application Manager:

ingress:
  enabled: true
  public:
    enabled: false
  admin:
    enabled: true
    hostname: "{{ .Values.flowx.ingress.admin }}"
    path: /appmanager(/|$)(.*)
    annotations:
      nginx.ingress.kubernetes.io/rewrite-target: /$2
      nginx.ingress.kubernetes.io/cors-allow-headers: DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization,flowx-platform

Note: Replace placeholders with actual values for your environment before starting the service.