Data-Sync job setup

Overview

The Data-Sync Job synchronizes data across multiple databases to maintain consistency and up-to-date information throughout your system. It operates by connecting to various databases, retrieving data, and synchronizing changes across them. The job logs all actions and can be scheduled to run at regular intervals.

Quick start

# 1. Configure your environment variables in a data-sync-job.yaml file
# 2. Apply the configuration
kubectl apply -f data-sync-job.yaml
# 3. Monitor the job status
kubectl get jobs
# 4. Check logs if needed
kubectl logs job/data-sync-job

Required environment variables

Database connections

The Data-Sync Job requires connection details for multiple databases. Configure the following sections based on your deployment.

MongoDB connections

Each MongoDB-based service requires the following variables:

Component	Required Variables
CMS	`FLOWX_DATASOURCE_CMS_URI`, `CMS_MONGO_USERNAME`, `CMS_MONGO_PASSWORD`, `CMS_MONGO_DATABASE`
Scheduler	`FLOWX_DATASOURCE_SCHEDULER_URI`, `SCHEDULER_MONGO_USERNAME`, `SCHEDULER_MONGO_PASSWORD`, `SCHEDULER_MONGO_DATABASE`
Task Manager	`FLOWX_DATASOURCE_TASKMANAGER_URI`, `TASKMANAGER_MONGO_USERNAME`, `TASKMANAGER_MONGO_PASSWORD`, `TASKMANAGER_MONGO_DATABASE`
Document Plugin	`FLOWX_DATASOURCE_DOCUMENTPLUGIN_URI`, `DOCUMENTPLUGIN_MONGO_USERNAME`, `DOCUMENTPLUGIN_MONGO_PASSWORD`, `DOCUMENTPLUGIN_MONGO_DATABASE`
Notification Plugin	`FLOWX_DATASOURCE_NOTIFICATIONPLUGIN_URI`, `NOTIFICATIONPLUGIN_MONGO_USERNAME`, `NOTIFICATIONPLUGIN_MONGO_PASSWORD`, `NOTIFICATIONPLUGIN_MONGO_DATABASE`
App Runtime	`FLOWX_DATASOURCE_APPRUNTIME_URI`, `APPRUNTIME_MONGO_USERNAME`, `APPRUNTIME_MONGO_PASSWORD`, `APPRUNTIME_MONGO_DATABASE`
Integration Designer	`FLOWX_DATASOURCE_INTEGRATIONDESIGNER_URI`, `INTEGRATIONDESIGNER_MONGO_USERNAME`, `INTEGRATIONDESIGNER_MONGO_PASSWORD`, `INTEGRATIONDESIGNER_MONGO_DATABASE`
Admin	`FLOWX_DATASOURCE_ADMIN_URI`, `ADMIN_MONGO_USERNAME`, `ADMIN_MONGO_PASSWORD`, `ADMIN_MONGO_DATABASE`

MongoDB URI format

mongodb://${USERNAME}:${PASSWORD}@mongodb-0.mongodb-headless,mongodb-1.mongodb-headless,mongodb-arbiter-0.mongodb-arbiter-headless:27017/${DATABASE}

PostgreSQL connections

Component	Required Variables
Process Engine	`FLOWX_DATASOURCE_ENGINE_URL`, `FLOWX_DATASOURCE_ENGINE_USERNAME`, `FLOWX_DATASOURCE_ENGINE_PASSWORD`, `FLOWX_DATASOURCE_ENGINE_DRIVERCLASSNAME`
Application Manager	`FLOWX_DATASOURCE_APPMANAGER_URL`, `FLOWX_DATASOURCE_APPMANAGER_USERNAME`, `FLOWX_DATASOURCE_APPMANAGER_PASSWORD`, `FLOWX_DATASOURCE_APPMANAGER_DRIVERCLASSNAME`
Authentication System	`FLOWX_DATASOURCE_AUTHSYSTEM_URL`, `FLOWX_DATASOURCE_AUTHSYSTEM_USERNAME`, `FLOWX_DATASOURCE_AUTHSYSTEM_PASSWORD`
Organization Manager	`FLOWX_DATASOURCE_ORGMANAGER_URL`, `FLOWX_DATASOURCE_ORGMANAGER_USERNAME`, `FLOWX_DATASOURCE_ORGMANAGER_PASSWORD`, `FLOWX_DATASOURCE_ORGMANAGER_DRIVERCLASSNAME`

Driver class names

PostgreSQL: org.postgresql.Driver
Oracle: oracle.jdbc.OracleDriver

Additional configuration

Variable	Description
`SPRING_JPA_DATABASE`	Database type for Spring JPA (e.g., `postgresql`, `oracle`)
`SPRING_JPA_PROPERTIES_HIBERNATE_DEFAULTSCHEMA`	Default schema for Hibernate
`LOGGING_CONFIG_FILE`	Path to logging configuration file

Service to database mapping

Each service in your environment corresponds to specific database datasources:

Service	Datasources
`scheduler-core`	scheduler
`cms-core`	cms
`task-management-plugin`	task-manager
`document-plugin`	document-plugin
`notification-plugin`	notification-plugin
`runtime-manager`	app-runtime, app-manager
`integration-designer`	integration-designer
`admin`	admin, engine
`process-engine`	engine
`application-manager`	app-manager
`authorization-system`	auth-system

Sample configuration

apiVersion: batch/v1
kind: Job
metadata:
  name: data-sync-job
spec:
  template:
    spec:
      containers:
      - name: data-sync
        image: your-registry/data-sync:latest
        env:
        # MongoDB connections
        - name: FLOWX_DATASOURCE_CMS_URI
          value: "mongodb://${CMS_MONGO_USERNAME}:${CMS_MONGO_PASSWORD}@mongodb-0.mongodb-headless:27017/${CMS_MONGO_DATABASE}"
        # Add all other required environment variables
      restartPolicy: Never
  backoffLimit: 3

SpiceDB configuration

Environment Variable	Description	Default Value
`FLOWX_SPICEDB_HOST`	SpiceDB server hostname	`spicedb`
`FLOWX_SPICEDB_PORT`	SpiceDB server port	`50051`
`FLOWX_SPICEDB_TOKEN`	SpiceDB authentication token	`-`

Best practices

Store sensitive credentials in Kubernetes Secrets and reference them in your deployment
Include the Data-Sync Job in your CI/CD pipeline for automated deployment
Schedule regular runs using a Kubernetes CronJob for periodic synchronization
Monitor job execution and set up alerts for failures

Troubleshooting

Database connection failures

Symptoms: The Data-Sync Job fails to start or exits with database connection errors.Solutions:

Verify MongoDB and PostgreSQL connection strings are correctly formatted
Check that database credentials are correct and the user has appropriate permissions
Ensure network connectivity between the job pod and database services
For MongoDB, confirm the replica set is healthy and reachable
For PostgreSQL, verify the JDBC URL format and driver class name

Sync failures

Symptoms: Data synchronization completes partially or fails.Note: The Data-Sync Job does not use Kafka. It is a batch job that connects directly to each service’s database to synchronize data.Solutions:

Check job logs for specific error messages: kubectl logs job/data-sync-job
Verify all database connection strings are correct and reachable from the job pod
Ensure database users have the required read/write permissions

Data inconsistencies after sync

Symptoms: Data across services appears out of date or mismatched after the job completes.Solutions:

Verify that the correct services are being synced
Check the service-to-database mapping to ensure each service points to the right datasource
Re-run the Data-Sync Job and monitor logs for partial failures
Confirm that no concurrent writes occurred during the sync window

Missing required variables

Symptoms: Job fails immediately on startup with configuration errors.Solutions:

Ensure all required environment variables are set for each database connection
Check for typos in environment variable names

Redis Configuration

Complete Redis setup including Sentinel and Cluster modes

Kafka Authentication

Configure Kafka security and authentication

IAM Configuration

Identity and access management setup

Microservices

Plugins

Observability

Access management

Data-Sync job setup

Overview

Quick start

Required environment variables

Database connections

MongoDB connections

MongoDB URI format

PostgreSQL connections

Driver class names

Additional configuration

Service to database mapping

Sample configuration

SpiceDB configuration

Best practices

Troubleshooting

Redis Configuration

Kafka Authentication

IAM Configuration

​Overview

​Quick start

​Required environment variables

​Database connections

​MongoDB connections

MongoDB URI format

​PostgreSQL connections

Driver class names

​Additional configuration

​Service to database mapping

​Sample configuration

​SpiceDB configuration

​Best practices

​Troubleshooting

​Related resources

Redis Configuration

Kafka Authentication

IAM Configuration

Overview

Quick start

Required environment variables

Database connections

MongoDB connections

PostgreSQL connections

Additional configuration

Service to database mapping

Sample configuration

SpiceDB configuration

Best practices

Troubleshooting

Related resources