> ## Documentation Index
> Fetch the complete documentation index at: https://docs.flowx.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Data-Sync job setup

> Comprehensive guide for configuring and deploying the Data-Sync Job in your Kubernetes environment

## Overview

The Data-Sync Job synchronizes data across multiple databases to maintain consistency and up-to-date information throughout your system. It operates by connecting to various databases, retrieving data, and synchronizing changes across them. The job logs all actions and can be scheduled to run at regular intervals.

## Quick Start

```bash theme={"system"}
# 1. Configure your environment variables in a data-sync-job.yaml file
# 2. Apply the configuration
kubectl apply -f data-sync-job.yaml
# 3. Monitor the job status
kubectl get jobs
# 4. Check logs if needed
kubectl logs job/data-sync-job
```

## Required environment variables

### Core configuration

| Variable         | Description                                             | Example                     |
| ---------------- | ------------------------------------------------------- | --------------------------- |
| `CONFIG_PROFILE` | Specifies which configuration profile to use (required) | `k8stemplate_v2,kafka-auth` |

### Database connections

The Data-Sync Job requires connection details for multiple databases. Configure the following sections based on your deployment.

#### MongoDB connections

Each MongoDB-based service requires the following variables:

| Component                | Required Variables                                                                                                                                           |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **CMS**                  | `FLOWX_DATASOURCE_CMS_URI`, `CMS_MONGO_USERNAME`, `CMS_MONGO_PASSWORD`, `CMS_MONGO_DATABASE`                                                                 |
| **Scheduler**            | `FLOWX_DATASOURCE_SCHEDULER_URI`, `SCHEDULER_MONGO_USERNAME`, `SCHEDULER_MONGO_PASSWORD`, `SCHEDULER_MONGO_DATABASE`                                         |
| **Task Manager**         | `FLOWX_DATASOURCE_TASKMANAGER_URI`, `TASKMANAGER_MONGO_USERNAME`, `TASKMANAGER_MONGO_PASSWORD`, `TASKMANAGER_MONGO_DATABASE`                                 |
| **Document Plugin**      | `FLOWX_DATASOURCE_DOCUMENTPLUGIN_URI`, `DOCUMENTPLUGIN_MONGO_USERNAME`, `DOCUMENTPLUGIN_MONGO_PASSWORD`, `DOCUMENTPLUGIN_MONGO_DATABASE`                     |
| **Notification Plugin**  | `FLOWX_DATASOURCE_NOTIFICATIONPLUGIN_URI`, `NOTIFICATIONPLUGIN_MONGO_USERNAME`, `NOTIFICATIONPLUGIN_MONGO_PASSWORD`, `NOTIFICATIONPLUGIN_MONGO_DATABASE`     |
| **App Runtime**          | `FLOWX_DATASOURCE_APPRUNTIME_URI`, `APPRUNTIME_MONGO_USERNAME`, `APPRUNTIME_MONGO_PASSWORD`, `APPRUNTIME_MONGO_DATABASE`                                     |
| **Integration Designer** | `FLOWX_DATASOURCE_INTEGRATIONDESIGNER_URI`, `INTEGRATIONDESIGNER_MONGO_USERNAME`, `INTEGRATIONDESIGNER_MONGO_PASSWORD`, `INTEGRATIONDESIGNER_MONGO_DATABASE` |
| **Admin**                | `FLOWX_DATASOURCE_ADMIN_URI`, `ADMIN_MONGO_USERNAME`, `ADMIN_MONGO_PASSWORD`, `ADMIN_MONGO_DATABASE`                                                         |

##### MongoDB URI format

```
mongodb://${USERNAME}:${PASSWORD}@mongodb-0.mongodb-headless,mongodb-1.mongodb-headless,mongodb-arbiter-0.mongodb-arbiter-headless:27017/${DATABASE}
```

#### PostgreSQL connections

| Component               | Required Variables                                                                                                                                               |
| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Process Engine**      | `FLOWX_DATASOURCE_ENGINE_URL`, `FLOWX_DATASOURCE_ENGINE_USERNAME`, `FLOWX_DATASOURCE_ENGINE_PASSWORD`, `FLOWX_DATASOURCE_ENGINE_DRIVERCLASSNAME`                 |
| **Application Manager** | `FLOWX_DATASOURCE_APPMANAGER_URL`, `FLOWX_DATASOURCE_APPMANAGER_USERNAME`, `FLOWX_DATASOURCE_APPMANAGER_PASSWORD`, `FLOWX_DATASOURCE_APPMANAGER_DRIVERCLASSNAME` |

##### Driver class names

* PostgreSQL: `org.postgresql.Driver`
* Oracle: `oracle.jdbc.OracleDriver`

### Additional configuration

| Variable                                        | Description                                                 |
| ----------------------------------------------- | ----------------------------------------------------------- |
| `SPRING_JPA_DATABASE`                           | Database type for Spring JPA (e.g., `postgresql`, `oracle`) |
| `SPRING_JPA_PROPERTIES_HIBERNATE_DEFAULTSCHEMA` | Default schema for Hibernate                                |
| `LOGGING_CONFIG_FILE`                           | Path to logging configuration file                          |

## Service to database mapping

Each service in your environment corresponds to specific database datasources:

| Service                  | Datasources             |
| ------------------------ | ----------------------- |
| `scheduler-core`         | scheduler               |
| `cms-core`               | cms                     |
| `task-management-plugin` | task-manager            |
| `document-plugin`        | document-plugin         |
| `notification-plugin`    | notification-plugin     |
| `runtime-manager`        | app-runtime, appmanager |
| `integration-designer`   | integration-designer    |
| `admin`                  | admin, engine           |
| `process-engine`         | engine                  |
| `application-manager`    | appmanager              |

## Sample configuration

```yaml theme={"system"}
apiVersion: batch/v1
kind: Job
metadata:
  name: data-sync-job
spec:
  template:
    spec:
      containers:
      - name: data-sync
        image: your-registry/data-sync:latest
        env:
        - name: CONFIG_PROFILE # deprecated since v4.7.6
          value: k8stemplate_v2,kafka-auth
        # MongoDB connections
        - name: FLOWX_DATASOURCE_CMS_URI
          value: "mongodb://${CMS_MONGO_USERNAME}:${CMS_MONGO_PASSWORD}@mongodb-0.mongodb-headless:27017/${CMS_MONGO_DATABASE}"
        # Add all other required environment variables
      restartPolicy: Never
  backoffLimit: 3
```

## Troubleshooting

### Common issues

1. **Database connection failures**:
   * Verify MongoDB and PostgreSQL connection strings
   * Check that database credentials are correct
   * Ensure network connectivity between the job pod and databases

2. **Missing Required Variables**:
   * Ensure all required environment variables are properly set
   * Check for typos in environment variable names

### Logs

Monitor the Data-Sync Job logs to diagnose issues:

```bash theme={"system"}
kubectl logs job/data-sync-job
```

## Best practices

1. Store sensitive credentials in Kubernetes Secrets and reference them in your deployment
2. Include the Data-Sync Job in your CI/CD pipeline for automated deployment
3. Schedule regular runs using a Kubernetes CronJob for periodic synchronization
4. Monitor job execution and set up alerts for failures

## Next steps

After successfully configuring and deploying the Data-Sync Job:

1. Verify data consistency across databases
2. Set up monitoring and alerting for job status
3. Consider automating deployment through your CI/CD pipeline