> ## Documentation Index
> Fetch the complete documentation index at: https://docs.flowx.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Scheduler setup

> This guide will walk you through the process of setting up the Scheduler service.

## Infrastructure prerequisites

* **MongoDB**
* **Kafka**
* **OpenID Connect Settings**

***

## Dependencies

* [MongoDB](https://www.mongodb.com/2) database
* Ability to connect to a Kafka instance used by the FlowX Engine
* Scheduler service account - required for using Start Timer event node. For more information, see the [scheduler service account configuration](./access-management/configuring-an-iam-solution#scheduler-service-account).

The service comes with most of the required configuration properties pre-filled. However, certain custom environment variables need to be set up.

<Warning>
  This service needs to connect to a Mongo database that has replicas to work correctly.
</Warning>

***

## Scheduler configuration

### Scheduler

```yaml theme={"system"}
scheduler:
  thread-count: 30  # Configure the number of threads to be used for sending expired messages.
  callbacks-thread-count: 60 # Configure the number of threads for handling Kafka responses, whether the message was successfully sent or not
  cronExpression: "*/10 * * * * *" #every 10 seconds
  retry: # new retry mechanism
    max-attempts: 3
    seconds: 1
    thread-count: 3
    cronExpression: "*/10 * * * * *" #every 10 seconds
  cleanup:
    cronExpression: "*/25 * * * * *" #every 25 seconds
```

* `SCHEDULER_THREAD_COUNT`: Used to configure the number of threads to be used for sending expired.
* `SCHEDULER_CALLBACKS_THREAD_COUNT`: Used to configure the number of threads for handling Kafka responses, whether the message was successfully sent or not.

<Info>
  The "scheduler.cleanup.cronExpression" is valid for both scheduler and timer event scheduler.
</Info>

#### Retry mechanism

* `SCHEDULER_RETRY_THREAD_COUNT`: Specify the number of threads to use for resending messages that need to be retried.
* `SCHEDULER_RETRY_MAX_ATTEMPTS`: This configuration parameter sets the number of retry attempts. For instance, if it's set to 3, it means that the system will make a maximum of three retry attempts for message resending.
* `SCHEDULER_RETRY_SECONDS`: This configuration parameter defines the time interval, in seconds, for retry attempts. For example, when set to 1, it indicates that the system will retry the operation after a one-second delay.

#### Cleanup

* `SCHEDULER_CLEANUP_CRONEXPRESSION`: It specifies how often, in seconds, events that have already been processed should be cleaned up from the database.

#### Recovery mechanism

```yaml theme={"system"}
flowx:
  timer-calculator:
    delay-max-repetitions: 1000000
```

<Tip>
  You have a "next execution" set for 10:25, and the cycle step is 10 minutes. If the instance goes down for 2 hours, the next execution time should be 12:25, not 10:35. To calculate this, you add 10 minutes repeatedly to 10:25 until you reach the current time. So, it would be 10:25 + 10 min + 10 min + 10 min, until you reach the current time of 12:25. This ensures that the next execution time is adjusted correctly after the downtime.
</Tip>

* `FLOWX_TIMER_CALCULATOR_DELAY_MAX_REPETITIONS`: This means that, for example, if our cycle step is set to one second and the system experiences a downtime of two weeks, which is equivalent to 1,209,600 seconds, and we have the "max repetitions" set to 1,000,000, it will attempt to calculate the next schedule. However, when it reaches the maximum repetitions, an exception is thrown, making it impossible to calculate the next schedule. As a result, the entry remains locked and needs to be rescheduled. This scenario represents a critical case where the system experiences extended downtime, and the cycle step is very short (e.g., 1 second), leading to the inability to determine the next scheduled event.

### Timer event scheduler

<Info>
  Configuration for Timer Event scheduler designed to manage timer events. Similar configuration to scheduler.
</Info>

```yaml theme={"system"}
timer-event-scheduler:
  thread-count: 30
  callbacks-thread-count: 60
  cronExpression: "*/1 * * * * *" #every 1 seconds
  retry:
    max-attempts: 3
    seconds: 1
    thread-count: 3
    cronExpression: "*/5 * * * * *" #every 5 seconds
```

***

## OpenID connect settings

| Environment variable                                                        | Description                                                                                                                                           | Default value                                                                                      |
| --------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| `SECURITY_TYPE`                                                             | Token validation mechanism (JWT public key validation)                                                                                                | `jwt-public-key`                                                                                   |
| `SECURITY_OAUTH2_BASESERVERURL`                                             | Base URL of the Keycloak server                                                                                                                       |                                                                                                    |
| `SECURITY_OAUTH2_SAREALM`                                                   | Service-accounts realm ID                                                                                                                             | `00000002-0002-4002-8002-000000000002`                                                             |
| `FLOWX_LIB_SECURITY_SERVICES_ORGANIZATIONMANAGER_BASEURL`                   | URL of the organization-manager service, used by the security library                                                                                 | `http://organization-manager:80`                                                                   |
| `SECURITY_PATHAUTHORIZATIONS_0_PATH`                                        | Security path pattern                                                                                                                                 | `/api/**`                                                                                          |
| `SECURITY_PATHAUTHORIZATIONS_0_ROLESALLOWED`                                | Roles allowed for path access                                                                                                                         | `ANY_AUTHENTICATED_USER`                                                                           |
| `SPRING_SECURITY_OAUTH2_CLIENT_REGISTRATION_MAINIDENTITY_CLIENTID`          | Service account client ID                                                                                                                             | `flowx-scheduler-core-sa`                                                                          |
| `SPRING_SECURITY_OAUTH2_CLIENT_REGISTRATION_MAINIDENTITY_CLIENTSECRET`      | Service account client secret (Keycloak-issued)                                                                                                       |                                                                                                    |
| `SPRING_SECURITY_OAUTH2_CLIENT_REGISTRATION_ANONYMOUSIDENTITY_CLIENTID`     | Anonymous service account client ID, used for [anonymous runtime access](/5.9/docs/platform-deep-dive/user-roles-management/anonymous-runtime-access) | `flowx-anonymous-sa`                                                                               |
| `SPRING_SECURITY_OAUTH2_CLIENT_REGISTRATION_ANONYMOUSIDENTITY_CLIENTSECRET` | Anonymous service account client secret                                                                                                               |                                                                                                    |
| `SPRING_SECURITY_OAUTH2_CLIENT_PROVIDER_MAINAUTHPROVIDER_TOKENURI`          | Provider token URI, resolved against the service-accounts realm                                                                                       | `${SECURITY_OAUTH2_BASESERVERURL}/realms/${SECURITY_OAUTH2_SAREALM}/protocol/openid-connect/token` |

<Warning>
  **Upgrading from 5.1.x?** Remove the legacy opaque-token env vars: `SECURITY_OAUTH2_REALM`, `SECURITY_OAUTH2_CLIENT_CLIENTID`, `SECURITY_OAUTH2_CLIENT_CLIENTSECRET`, and `SECURITY_OAUTH2_SERVICEACCOUNT_ADMIN_*`. These belong to the removed introspection model and prevent the service from starting on 5.9.x. See the [authentication and IAM migration guide](/5.9/migrating-from-5.1-lts/authentication-iam) for the full list.
</Warning>

<Info>
  When deploying with the FlowX Helm chart, `SECURITY_OAUTH2_BASESERVERURL` is supplied through the chart value `flowx.keycloak.baseServerUrl` and the service-account client secrets are injected from the chart-managed Keycloak secret. The remaining values ship as image defaults.
</Info>

The service account is essential for enabling the [**Start Timer**](../docs/building-blocks/node/timer-events/timer-start-event) event node.

More details about the necessary service account, here:

[Scheduler service account](./access-management/configuring-an-iam-solution#scheduler-service-account)

***

## Configuring datasoruce (MongoDB)

The MongoDB database is used to persist scheduled messages until they are sent back. The following configurations need to be set using environment variables:

* `SPRING_DATA_MONGODB_URI`: The URI for the MongoDB database.

***

## Configuring Kafka

### Core Kafka settings

| Environment Variable               | Description                                                                          | Default Value        |
| ---------------------------------- | ------------------------------------------------------------------------------------ | -------------------- |
| `KAFKA_BOOTSTRAP_SERVERS`          | Kafka broker addresses (fallback: `SPRING_KAFKA_BOOTSTRAP_SERVERS`)                  | `localhost:9092`     |
| `KAFKA_SECURITY_PROTOCOL`          | Security protocol for Kafka connections (fallback: `SPRING_KAFKA_SECURITY_PROTOCOL`) | `PLAINTEXT`          |
| `SPRING_KAFKA_CONSUMER_GROUPID`    | Consumer group identifier                                                            | `scheduler-consumer` |
| `KAFKA_MESSAGE_MAX_BYTES`          | Maximum message size (bytes)                                                         | `52428800` (50 MB)   |
| `KAFKA_AUTHEXCEPTIONRETRYINTERVAL` | Retry interval after authorization exceptions (seconds)                              | `10`                 |

### Consumer configuration

| Environment Variable                              | Description                                 | Default Value                 |
| ------------------------------------------------- | ------------------------------------------- | ----------------------------- |
| `KAFKA_CONSUMER_THREADS`                          | Number of Kafka consumer threads            | `1`                           |
| `KAFKA_CONSUMER_SCHEDULEDTIMEREVENTS_THREADS`     | Number of threads for starting Timer Events | `1`                           |
| `KAFKA_CONSUMER_SCHEDULEDTIMEREVENTS_GROUPID`     | Consumer group for starting timer events    | `scheduled-timer-events`      |
| `KAFKA_CONSUMER_STOPSCHEDULEDTIMEREVENTS_THREADS` | Number of threads for stopping Timer Events | `1`                           |
| `KAFKA_CONSUMER_STOPSCHEDULEDTIMEREVENTS_GROUPID` | Consumer group for stopping timer events    | `stop-scheduled-timer-events` |

### OAuth authentication (when using SASL\_PLAINTEXT)

| Environment Variable             | Description          | Default Value          |
| -------------------------------- | -------------------- | ---------------------- |
| `KAFKA_OAUTH_CLIENT_ID`          | OAuth client ID      | `kafka`                |
| `KAFKA_OAUTH_CLIENT_SECRET`      | OAuth client secret  | `kafka-secret`         |
| `KAFKA_OAUTH_TOKEN_ENDPOINT_URI` | OAuth token endpoint | `kafka.auth.localhost` |

<Info>
  When using the `kafka-auth` profile, the security protocol will automatically be set to `SASL_PLAINTEXT` and the SASL mechanism will be set to `OAUTHBEARER`.
</Info>

### Topic naming configuration

| Environment Variable             | Description                         | Default Value |
| -------------------------------- | ----------------------------------- | ------------- |
| `KAFKA_TOPIC_NAMING_PACKAGE`     | Package prefix for topic names      | `ai.flowx.`   |
| `KAFKA_TOPIC_NAMING_ENVIRONMENT` | Environment segment for topic names | ` `           |
| `KAFKA_TOPIC_NAMING_VERSION`     | Version suffix for topic names      | `.v1`         |
| `KAFKA_TOPIC_NAMING_SEPARATOR`   | Primary separator for topic names   | `.`           |
| `KAFKA_TOPIC_NAMING_SEPARATOR2`  | Secondary separator for topic names | `-`           |

### Kafka topics

#### Schedule topics

| Environment Variable           | Description                                      | Default Value                            |
| ------------------------------ | ------------------------------------------------ | ---------------------------------------- |
| `KAFKA_TOPIC_SCHEDULE_IN_SET`  | Receives scheduled message setting requests      | `ai.flowx.core.trigger.set.schedule.v1`  |
| `KAFKA_TOPIC_SCHEDULE_IN_STOP` | Handles requests to terminate scheduled messages | `ai.flowx.core.trigger.stop.schedule.v1` |

#### Timer events topics

| Environment Variable                       | Description                     | Default Value                                        |
| ------------------------------------------ | ------------------------------- | ---------------------------------------------------- |
| `KAFKA_TOPIC_SCHEDULEDTIMEREVENTS_IN_SET`  | Topic for setting timer events  | `ai.flowx.core.trigger.set.timer-event-schedule.v1`  |
| `KAFKA_TOPIC_SCHEDULEDTIMEREVENTS_IN_STOP` | Topic for stopping timer events | `ai.flowx.core.trigger.stop.timer-event-schedule.v1` |

<Info>
  Make sure the topics configured for this service don't follow the engine pattern.
</Info>

***

## Configuring logging

The following environment variables can be set to control log levels:

* `LOGGING_LEVEL_ROOT`: Root Spring Boot microservice logs (Default: `INFO`)
* `LOGGING_LEVEL_APP`: App level logs (Default: `INFO`)

***

## Troubleshooting

### Common issues

<AccordionGroup>
  <Accordion title="Scheduled processes not triggering">
    **Symptoms:** Scheduled processes do not start at the expected times.

    **Solutions:**

    1. Verify the cron expressions in `SCHEDULER_CLEANUP_CRONEXPRESSION` and the scheduler `cronExpression` settings are correct
    2. Check that Kafka topics (`KAFKA_TOPIC_SCHEDULE_IN_SET`, `KAFKA_TOPIC_SCHEDULE_IN_STOP`) are correctly configured and accessible
    3. Confirm the scheduler service is healthy by checking `/actuator/health`
    4. Review scheduler logs at `DEBUG` level for missed or skipped executions
  </Accordion>

  <Accordion title="Duplicate schedule executions">
    **Symptoms:** Scheduled messages are sent multiple times for the same event.

    **Solutions:**

    1. Check the replica count — running multiple replicas without proper leader election can cause duplicates
    2. Verify that the MongoDB replica set is healthy, as the scheduler relies on it for distributed locking
    3. Review `SCHEDULER_THREAD_COUNT` and `SCHEDULER_CALLBACKS_THREAD_COUNT` to ensure they are not excessively high for your workload
  </Accordion>

  <Accordion title="Scheduler fails to start">
    **Symptoms:** The scheduler service crashes or fails during startup.

    **Solutions:**

    1. Verify MongoDB connectivity and ensure the database has replicas enabled (required for the scheduler to work correctly)
    2. Check that Kafka bootstrap servers are reachable (`KAFKA_BOOTSTRAP_SERVERS`)
    3. Confirm the service account credentials are valid — verify `SPRING_SECURITY_OAUTH2_CLIENT_REGISTRATION_MAINIDENTITY_CLIENTID` and `SPRING_SECURITY_OAUTH2_CLIENT_REGISTRATION_MAINIDENTITY_CLIENTSECRET`
    4. Review startup logs for specific connection error messages
  </Accordion>

  <Accordion title="Timer events not firing">
    **Symptoms:** Timer event nodes in processes do not trigger at the configured times.

    **Solutions:**

    1. Verify the timer event configuration in the process definition (Start Timer, Intermediate Timer)
    2. Check that the timer event Kafka topics are correctly set (`KAFKA_TOPIC_SCHEDULEDTIMEREVENTS_IN_SET`, `KAFKA_TOPIC_SCHEDULEDTIMEREVENTS_IN_STOP`)
    3. Ensure the scheduler service account has the necessary permissions — see the [Scheduler service account](./access-management/configuring-an-iam-solution#scheduler-service-account) configuration
    4. Review the `FLOWX_TIMER_CALCULATOR_DELAY_MAX_REPETITIONS` setting if the system experienced recent downtime
  </Accordion>
</AccordionGroup>

***

## Related resources

<CardGroup cols={2}>
  <Card title="Scheduled Processes" icon="clock" href="../docs/projects/runtime/scheduled-processes">
    Configure and manage scheduled process executions
  </Card>

  <Card title="Timer Events" icon="timer" href="../docs/building-blocks/node/timer-events/timer-events">
    Timer event node types and configuration options
  </Card>

  <Card title="Redis Configuration" icon="database" href="./redis-configuration">
    Complete Redis setup including Sentinel and Cluster modes
  </Card>
</CardGroup>