Data-Sync job setup

​

Overview

The Data-Sync Job synchronizes data across multiple databases to maintain consistency and up-to-date information throughout your system. It operates by connecting to various databases, retrieving data, and synchronizing changes across them. The job logs all actions and can be scheduled to run at regular intervals.

​

Quick Start

# 1. Configure your environment variables in a data-sync-job.yaml file
# 2. Apply the configuration
kubectl apply -f data-sync-job.yaml
# 3. Monitor the job status
kubectl get jobs
# 4. Check logs if needed
kubectl logs job/data-sync-job

​

Required environment variables

​

Core configuration

⚠️ Warning: Do not include spaces in the FLOWX_SKIPPEDRESOURCESERVICES value.

​

Database connections

The Data-Sync Job requires connection details for multiple databases. Configure the following sections based on your deployment.

​

MongoDB connections

Each MongoDB-based service requires the following variables:

MongoDB URI format

mongodb://${USERNAME}:${PASSWORD}@mongodb-0.mongodb-headless,mongodb-1.mongodb-headless,mongodb-arbiter-0.mongodb-arbiter-headless:27017/${DATABASE}

​

PostgreSQL connections

Driver class names

• PostgreSQL: org.postgresql.Driver
• Oracle: oracle.jdbc.OracleDriver

​

Additional configuration

​

Service to database mapping

Each service in your environment corresponds to specific database datasources:

​

Sample configuration

apiVersion: batch/v1
kind: Job
metadata:
  name: data-sync-job
spec:
  template:
    spec:
      containers:
      - name: data-sync
        image: your-registry/data-sync:latest
        env:
        - name: CONFIG_PROFILE
          value: k8stemplate_v2,kafka-auth
        - name: FLOWX_SKIPPEDRESOURCESERVICES
          value: "document-plugin,notification-plugin"
        # MongoDB connections
        - name: FLOWX_DATASOURCE_CMS_URI
          value: "mongodb://${CMS_MONGO_USERNAME}:${CMS_MONGO_PASSWORD}@mongodb-0.mongodb-headless:27017/${CMS_MONGO_DATABASE}"
        # Add all other required environment variables
      restartPolicy: Never
  backoffLimit: 3

​

Troubleshooting

​

Common issues

1. Database connection failures:

   • Verify MongoDB and PostgreSQL connection strings
   • Check that database credentials are correct
   • Ensure network connectivity between the job pod and databases

2. Missing Required Variables:

   • Ensure all required environment variables are properly set
   • Check for typos in environment variable names

3. Service Synchronization Failures:

   • If a service isn’t installed but data-sync is attempting to sync it, add it to FLOWX_SKIPPEDRESOURCESERVICES

​

Logs

Monitor the Data-Sync Job logs to diagnose issues:

kubectl logs job/data-sync-job

​

Best practices

1. Store sensitive credentials in Kubernetes Secrets and reference them in your deployment
2. Include the Data-Sync Job in your CI/CD pipeline for automated deployment
3. Schedule regular runs using a Kubernetes CronJob for periodic synchronization
4. Monitor job execution and set up alerts for failures

​

Next steps

After successfully configuring and deploying the Data-Sync Job:

1. Verify data consistency across databases
2. Set up monitoring and alerting for job status
3. Consider automating deployment through your CI/CD pipeline