​

Overview

The Data-Sync Job is designed to synchronize and transfer data across multiple databases, ensuring data consistency and up-to-date information across all connected systems.

It operates by connecting to multiple databases, retrieves data, and synchronizes changes across them. The job logs its actions and can be scheduled to run regularly, keeping all databases in sync and up-to-date.

​

Data-sync job setup guide

This guide details the essential environment variables required to configure and run the Data-Sync Job for synchronizing data across various databases.

​

Required environment variables

​

Config profile

• CONFIG_PROFILE - This environment variable must be set explicitly and exactly to ensure that no unintended profiles are loaded by mistake. The value of this variable should represent a minimal configuration state, relying only on defaults specified in the application.properties file of the application.

Example:

- name: CONFIG_PROFILE
  value: k8stemplate_v2,kafka-auth

​

Skipping services for data-sync

If your deployment doesn’t have certain plugins or other microservices installed, you should configure data-sync to skip migrations for those:

• FLOWX_SKIPPEDRESOURCESERVICES - Comma-separated list of service names for which data-sync should skip running migrations.

Example:

# Skip data-sync for document and notification plugins
- name: FLOWX_SKIPPEDRESOURCESERVICES
  value: "document-plugin,notification-plugin"

This prevents data-sync from failing when trying to synchronize databases for services that aren’t installed in your environment.

​

MongoDB connections

​

CMS MongoDB

• FLOWX_DATASOURCE_CMS_URI - MongoDB URI for CMS database.
  • Example: "mongodb://${CMS_MONGO_USERNAME}:${CMS_MONGO_PASSWORD}@mongodb-0.mongodb-headless,mongodb-1.mongodb-headless,mongodb-arbiter-0.mongodb-arbiter-headless:27017/${CMS_MONGO_DATABASE}"
• CMS_MONGO_USERNAME - Username for MongoDB CMS database.
• CMS_MONGO_PASSWORD - Password for the CMS MongoDB database.
• CMS_MONGO_DATABASE - Database name for CMS.

​

Scheduler MongoDB

• FLOWX_DATASOURCE_SCHEDULER_URI - MongoDB URI for Scheduler database.
  • Example: "mongodb://${SCHEDULER_MONGO_USERNAME}:${SCHEDULER_MONGO_PASSWORD}@mongodb-0.mongodb-headless,mongodb-1.mongodb-headless,mongodb-arbiter-0.mongodb-arbiter-headless:27017/${SCHEDULER_MONGO_DATABASE}"
• SCHEDULER_MONGO_USERNAME - Username for MongoDB Scheduler database.
• SCHEDULER_MONGO_PASSWORD - Password for the Scheduler MongoDB database.
• SCHEDULER_MONGO_DATABASE - Database name for Scheduler.

​

Task Manager MongoDB

• FLOWX_DATASOURCE_TASKMANAGER_URI - MongoDB URI for Task Manager database.
  • Example: "mongodb://${TASKMANAGER_MONGO_USERNAME}:${TASKMANAGER_MONGO_PASSWORD}@mongodb-0.mongodb-headless,mongodb-1.mongodb-headless,mongodb-arbiter-0.mongodb-arbiter-headless:27017/${TASKMANAGER_MONGO_DATABASE}"
• TASKMANAGER_MONGO_USERNAME - Username for MongoDB Task Manager database.
• TASKMANAGER_MONGO_PASSWORD - Password for the Task Manager MongoDB database.
• TASKMANAGER_MONGO_DATABASE - Database name for Task Manager.

​

Document plugin MongoDB

• FLOWX_DATASOURCE_DOCUMENTPLUGIN_URI - MongoDB URI for Document Plugin database.
  • Example: "mongodb://${DOCUMENTPLUGIN_MONGO_USERNAME}:${DOCUMENTPLUGIN_MONGO_PASSWORD}@mongodb-0.mongodb-headless,mongodb-1.mongodb-headless,mongodb-arbiter-0.mongodb-arbiter-headless:27017/${DOCUMENTPLUGIN_MONGO_DATABASE}"
• DOCUMENTPLUGIN_MONGO_DATABASE - MongoDB database name for the Document Plugin.
• DOCUMENTPLUGIN_MONGO_USERNAME - MongoDB username for the Document Plugin.
• DOCUMENTPLUGIN_MONGO_PASSWORD - Password for the MongoDB Document Plugin database.

​

Notification plugin MongoDB

• FLOWX_DATASOURCE_NOTIFICATIONPLUGIN_URI - MongoDB URI for the Notification Plugin.
  • Example: "mongodb://${NOTIFICATIONPLUGIN_MONGO_USERNAME}:${NOTIFICATIONPLUGIN_MONGO_PASSWORD}@mongodb-0.mongodb-headless,mongodb-1.mongodb-headless,mongodb-arbiter-0.mongodb-arbiter-headless:27017/${NOTIFICATIONPLUGIN_MONGO_DATABASE}"
• NOTIFICATIONPLUGIN_MONGO_USERNAME - MongoDB username for the Notification Plugin.
• NOTIFICATIONPLUGIN_MONGO_PASSWORD - Password for the MongoDB Notification Plugin database.
• NOTIFICATIONPLUGIN_MONGO_DATABASE - MongoDB database name for the Notification Plugin.

​

App Runtime MongoDB

• FLOWX_DATASOURCE_APPRUNTIME_URI - MongoDB URI for App Runtime database.
  • Example: "mongodb://${APPRUNTIME_MONGO_USERNAME}:${APPRUNTIME_MONGO_PASSWORD}@mongodb-0.mongodb-headless,mongodb-1.mongodb-headless,mongodb-arbiter-0.mongodb-arbiter-headless:27017/${APPRUNTIME_MONGO_DATABASE}"
• APPRUNTIME_MONGO_USERNAME - Username for MongoDB App Runtime database.
• APPRUNTIME_MONGO_DATABASE - Database name for App Runtime.

​

Integration Designer MongoDB

• FLOWX_DATASOURCE_INTEGRATIONDESIGNER_URI - MongoDB URI for Integration Designer database.
  • Example: "mongodb://${INTEGRATIONDESIGNER_MONGO_USERNAME}:${INTEGRATIONDESIGNER_MONGO_PASSWORD}@mongodb-0.mongodb-headless,mongodb-1.mongodb-headless,mongodb-arbiter-0.mongodb-arbiter-headless:27017/${INTEGRATIONDESIGNER_MONGO_DATABASE}"
• INTEGRATIONDESIGNER_MONGO_USERNAME - Username for MongoDB Integration Designer database.
• INTEGRATIONDESIGNER_MONGO_DATABASE - Database name for Integration Designer.

​

Admin MongoDB

• FLOWX_DATASOURCE_ADMIN_URI - MongoDB URI for Admin database.
  • Example: "mongodb://${ADMIN_MONGO_USERNAME}:${ADMIN_MONGO_PASSWORD}@mongodb-0.mongodb-headless,mongodb-1.mongodb-headless,mongodb-arbiter-0.mongodb-arbiter-headless:27017/${ADMIN_MONGO_DATABASE}"
• ADMIN_MONGO_USERNAME - Username for MongoDB Admin database.
• ADMIN_MONGO_DATABASE - Database name for Admin.

​

PostgreSQL connections

​

Process Engine database

• FLOWX_DATASOURCE_ENGINE_URL - PostgreSQL URL for Process Engine database.
• FLOWX_DATASOURCE_ENGINE_USERNAME - Username for PostgreSQL Process Engine database.
• FLOWX_DATASOURCE_ENGINE_DRIVERCLASSNAME - Driver class name for the Process Engine PostgreSQL database.

​

Application Manager database

• FLOWX_DATASOURCE_APPMANAGER_URL - PostgreSQL URL for Application Manager database.
• FLOWX_DATASOURCE_APPMANAGER_USERNAME - Username for PostgreSQL Application Manager database.
• FLOWX_DATASOURCE_APPMANAGER_PASSWORD - Password for the Application Manager PostgreSQL database.
• FLOWX_DATASOURCE_APPMANAGER_DRIVERCLASSNAME
  • Details: Required to ensure proper database connectivity. This value can be overridden for other databases, such as Oracle.
    • For PostgreSQL: org.postgresql.Driver
    • For Oracle: oracle.jdbc.OracleDriver

​

Spring JPA Variables

SPRING_JPA_DATABASE - The database type for Spring JPA (e.g., oracle). SPRING_JPA_PROPERTIES_HIBERNATE_DEFAULTSCHEMA - The default schema for Hibernate in Spring JPA.

​

Logging

• LOGGING_CONFIG_FILE - Configuration file for logging.

​

Deployment

To deploy the Data-Sync Job, apply the YAML configuration with the required environment variables:

kubectl apply -f data-sync-job.yaml

Monitor the Job status and logs as needed to ensure successful execution.