​
Overview

The Data-Sync Job synchronizes data across multiple databases to maintain consistency and up-to-date information throughout your system. It operates by connecting to various databases, retrieving data, and synchronizing changes across them. The job logs all actions and can be scheduled to run at regular intervals.

​
Quick Start

# 1. Configure your environment variables in a data-sync-job.yaml file
# 2. Apply the configuration
kubectl apply -f data-sync-job.yaml
# 3. Monitor the job status
kubectl get jobs
# 4. Check logs if needed
kubectl logs job/data-sync-job

​
Required environment variables

​
Core configuration

VariableDescriptionExample
CONFIG_PROFILESpecifies which configuration profile to use (required)k8stemplate_v2,kafka-auth
FLOWX_SKIPPEDRESOURCESERVICESComma-separated list of services to skip during synchronizationdocument-plugin,notification-plugin
⚠️ Warning: Do not include spaces in the FLOWX_SKIPPEDRESOURCESERVICES value.

​
Database connections

The Data-Sync Job requires connection details for multiple databases. Configure the following sections based on your deployment.

​
MongoDB connections

Each MongoDB-based service requires the following variables:
ComponentRequired Variables
CMSFLOWX_DATASOURCE_CMS_URI, CMS_MONGO_USERNAME, CMS_MONGO_PASSWORD, CMS_MONGO_DATABASE
SchedulerFLOWX_DATASOURCE_SCHEDULER_URI, SCHEDULER_MONGO_USERNAME, SCHEDULER_MONGO_PASSWORD, SCHEDULER_MONGO_DATABASE
Task ManagerFLOWX_DATASOURCE_TASKMANAGER_URI, TASKMANAGER_MONGO_USERNAME, TASKMANAGER_MONGO_PASSWORD, TASKMANAGER_MONGO_DATABASE
Document PluginFLOWX_DATASOURCE_DOCUMENTPLUGIN_URI, DOCUMENTPLUGIN_MONGO_USERNAME, DOCUMENTPLUGIN_MONGO_PASSWORD, DOCUMENTPLUGIN_MONGO_DATABASE
Notification PluginFLOWX_DATASOURCE_NOTIFICATIONPLUGIN_URI, NOTIFICATIONPLUGIN_MONGO_USERNAME, NOTIFICATIONPLUGIN_MONGO_PASSWORD, NOTIFICATIONPLUGIN_MONGO_DATABASE
App RuntimeFLOWX_DATASOURCE_APPRUNTIME_URI, APPRUNTIME_MONGO_USERNAME, APPRUNTIME_MONGO_PASSWORD, APPRUNTIME_MONGO_DATABASE
Integration DesignerFLOWX_DATASOURCE_INTEGRATIONDESIGNER_URI, INTEGRATIONDESIGNER_MONGO_USERNAME, INTEGRATIONDESIGNER_MONGO_PASSWORD, INTEGRATIONDESIGNER_MONGO_DATABASE
AdminFLOWX_DATASOURCE_ADMIN_URI, ADMIN_MONGO_USERNAME, ADMIN_MONGO_PASSWORD, ADMIN_MONGO_DATABASE
MongoDB URI format
mongodb://${USERNAME}:${PASSWORD}@mongodb-0.mongodb-headless,mongodb-1.mongodb-headless,mongodb-arbiter-0.mongodb-arbiter-headless:27017/${DATABASE}

​
PostgreSQL connections

ComponentRequired Variables
Process EngineFLOWX_DATASOURCE_ENGINE_URL, FLOWX_DATASOURCE_ENGINE_USERNAME, FLOWX_DATASOURCE_ENGINE_PASSWORD, FLOWX_DATASOURCE_ENGINE_DRIVERCLASSNAME
Application ManagerFLOWX_DATASOURCE_APPMANAGER_URL, FLOWX_DATASOURCE_APPMANAGER_USERNAME, FLOWX_DATASOURCE_APPMANAGER_PASSWORD, FLOWX_DATASOURCE_APPMANAGER_DRIVERCLASSNAME
Driver class names
  • PostgreSQL: org.postgresql.Driver
  • Oracle: oracle.jdbc.OracleDriver

​
Additional configuration

VariableDescription
SPRING_JPA_DATABASEDatabase type for Spring JPA (e.g., postgresql, oracle)
SPRING_JPA_PROPERTIES_HIBERNATE_DEFAULTSCHEMADefault schema for Hibernate
LOGGING_CONFIG_FILEPath to logging configuration file

​
Service to database mapping

Each service in your environment corresponds to specific database datasources:
ServiceDatasources
scheduler-corescheduler
cms-corecms
task-management-plugintask-manager
document-plugindocument-plugin
notification-pluginnotification-plugin
runtime-managerapp-runtime, appmanager
integration-designerintegration-designer
adminadmin, engine
process-engineengine
application-managerappmanager

​
Sample configuration

apiVersion: batch/v1
kind: Job
metadata:
  name: data-sync-job
spec:
  template:
    spec:
      containers:
      - name: data-sync
        image: your-registry/data-sync:latest
        env:
        - name: CONFIG_PROFILE # deprecated since v4.7.6
          value: k8stemplate_v2,kafka-auth
        - name: FLOWX_SKIPPEDRESOURCESERVICES
          value: "document-plugin,notification-plugin"
        # MongoDB connections
        - name: FLOWX_DATASOURCE_CMS_URI
          value: "mongodb://${CMS_MONGO_USERNAME}:${CMS_MONGO_PASSWORD}@mongodb-0.mongodb-headless:27017/${CMS_MONGO_DATABASE}"
        # Add all other required environment variables
      restartPolicy: Never
  backoffLimit: 3

​
Troubleshooting

​
Common issues

  1. Database connection failures:
    • Verify MongoDB and PostgreSQL connection strings
    • Check that database credentials are correct
    • Ensure network connectivity between the job pod and databases
  2. Missing Required Variables:
    • Ensure all required environment variables are properly set
    • Check for typos in environment variable names
  3. Service Synchronization Failures:
    • If a service isn’t installed but data-sync is attempting to sync it, add it to FLOWX_SKIPPEDRESOURCESERVICES

​
Logs

Monitor the Data-Sync Job logs to diagnose issues: 
kubectl logs job/data-sync-job

​
Best practices

  1. Store sensitive credentials in Kubernetes Secrets and reference them in your deployment
  2. Include the Data-Sync Job in your CI/CD pipeline for automated deployment
  3. Schedule regular runs using a Kubernetes CronJob for periodic synchronization
  4. Monitor job execution and set up alerts for failures

​
Next steps

After successfully configuring and deploying the Data-Sync Job:
  1. Verify data consistency across databases
  2. Set up monitoring and alerting for job status
  3. Consider automating deployment through your CI/CD pipeline