Scheduler setup
This guide will walk you through the process of setting up the Scheduler service.
Infrastructure prerequisites
- MongoDB: Version 4.4 or higher for storing taxonomies and contents.
- Kafka: Version 2.8 or higher.
- OpenID Connect Settings: Default settings are for Keycloak.
Dependencies
- MongoDB database
- Ability to connect to a Kafka instance used by the FlowX Engine
- Scheduler service account - required for using Start Timer event node - see here
The service comes with most of the required configuration properties pre-filled. However, certain custom environment variables need to be set up.
Dependencies
MongoDB helm example
Basic MongoDB configuration - helm values.yaml
scheduler-mdb:
existingSecret: {{secretName}}
mongodbDatabase: {{SchedulerDatabaseName}}
mongodbUsername: {{SchedulerDatabaseUser}}
persistence:
enabled: true
mountPath: /bitnami/mongodb
size: 4Gi
replicaSet:
enabled: true
name: rs0
pdb:
enabled: trues
minAvailable:
arbiter: 1
secondary: 1
replicas:
arbiter: 1
secondary: 1
useHostnames: true
serviceAccount:
create: false
usePassword: true
This service needs to connect to a Mongo database that has replicas, in order to work correctly.
Scheduler configuration
Scheduler
scheduler:
thread-count: 30 # Configure the number of threads to be used for sending expired messages.
callbacks-thread-count: 60 # Configure the number of threads for handling Kafka responses, whether the message was successfully sent or not
cronExpression: "*/10 * * * * *" #every 10 seconds
retry: # new retry mechanism
max-attempts: 3
seconds: 1
thread-count: 3
cronExpression: "*/10 * * * * *" #every 10 seconds
cleanup:
cronExpression: "*/25 * * * * *" #every 25 seconds
SCHEDULER_THREAD_COUNT
: Used to configure the number of threads to be used for sending expired.SCHEDULER_CALLBACKS_THREAD_COUNT
: Used to configure the number of threads for handling Kafka responses, whether the message was successfully sent or not.
The “scheduler.cleanup.cronExpression” is valid for both scheduler and timer event scheduler.
Retry mechanism
SCHEDULER_RETRY_THREAD_COUNT
: Specify the number of threads to use for resending messages that need to be retried.SCHEDULER_RETRY_MAX_ATTEMPTS
: This configuration parameter sets the number of retry attempts. For instance, if it’s set to 3, it means that the system will make a maximum of three retry attempts for message resending.SCHEDULER_RETRY_SECONDS
: This configuration parameter defines the time interval, in seconds, for retry attempts. For example, when set to 1, it indicates that the system will retry the operation after a one-second delay.
Cleanup
SCHEDULER_CLEANUP_CRONEXPRESSION
: It specifies how often, in seconds, events that have already been processed should be cleaned up from the database.
Recovery mechanism
flowx:
timer-calculator:
delay-max-repetitions: 1000000
You have a “next execution” set for 10:25, and the cycle step is 10 minutes. If the instance goes down for 2 hours, the next execution time should be 12:25, not 10:35. To calculate this, you add 10 minutes repeatedly to 10:25 until you reach the current time. So, it would be 10:25 + 10 min + 10 min + 10 min, until you reach the current time of 12:25. This ensures that the next execution time is adjusted correctly after the downtime.
FLOWX_TIMER_CALCULATOR_DELAY_MAX_REPETITIONS
: This means that, for example, if our cycle step is set to one second and the system experiences a downtime of two weeks, which is equivalent to 1,209,600 seconds, and we have the “max repetitions” set to 1,000,000, it will attempt to calculate the next schedule. However, when it reaches the maximum repetitions, an exception is thrown, making it impossible to calculate the next schedule. As a result, the entry remains locked and needs to be rescheduled. This scenario represents a critical case where the system experiences extended downtime, and the cycle step is very short (e.g., 1 second), leading to the inability to determine the next scheduled event.
Timer event scheduler
Configuration for Timer Event scheduler designed to manage timer events. Similar configuration to scheduler.
timer-event-scheduler:
thread-count: 30
callbacks-thread-count: 60
cronExpression: "*/1 * * * * *" #every 1 seconds
retry:
max-attempts: 3
seconds: 1
thread-count: 3
cronExpression: "*/5 * * * * *" #every 5 seconds
OpenID connect settings
Default settings are for Keycloak.
SECURITY_TYPE
: Indicates that OAuth 2.0 is the chosen security type, default value:oauth2
.SECURITY_PATHAUTHORIZATIONS_0_PATH
: Defines a security path or endpoint pattern. It specifies that the security settings apply to all paths under the “/api/” path. The**
is a wildcard that means it includes all subpaths under “/api/**“.SECURITY_PATHAUTHORIZATIONS_0_ROLESALLOWED
: Specifies the roles allowed for accessing the specified path. In this case, the roles allowed are empty (""). This might imply that access to the “/api/**” paths is open to all users or that no specific roles are required for authorization.
security:
type: oauth2
pathAuthorizations:
- path: "/api/**"
rolesAllowed: "ANY_AUTHENTICATED_USER"
SECURITY_OAUTH2_BASE_SERVER_URL
: This setting specifies the base URL of the OpenID server, which is used for authentication and authorization.SECURITY_OAUTH2_SERVICE_ACCOUNT_ADMIN_CLIENT_ID
: This setting specifies the service account that is essential for enabling the Start Timer event node. Ensure that you provide the correct client ID for this service account.SECURITY_OAUTH2_SERVICE_ACCOUNT_ADMIN_CLIENT_SECRET
: Along with the client ID, you must also specify the client secret associated with the service account for proper authentication.
More details about the necessary service account, here:
oauth2:
base-server-url: http://localhost:8080/auth
realm: flowx
client:
access-token-uri: ${security.oauth2.base-server-url}/realms/${security.oauth2.realm}/protocol/openid-connect/token
client-id: flowx-platform-authorize
client-secret: wrongsecret
resource:
user-info-uri: ${security.oauth2.base-server-url}/realms/${security.oauth2.realm}/protocol/openid-connect/userinfo
service-account:
admin:
client-id: flowx-scheduler-core-sa
client-secret: wrongsecret
Configuring datasoruce (MongoDB)
The MongoDB database is used to persist scheduled messages until they are sent back. The following configurations need to be set using environment variables:
SPRING_DATA_MONGODB_URI
: The URI for the MongoDB database.
Configuring Kafka
The following Kafka related configurations can be set by using environment variables:
SPRING_KAFKA_BOOTSTRAP_SERVERS
: Address of the Kafka serverSPRING_KAFKA_CONSUMER_GROUP_ID
: Group of consumersKAFKA_CONSUMER_THREADS
(default: 1): The number of Kafka consumer threadsKAFKA_CONSUMER_SCHEDULED_TIMER_EVENTS_THREADS
(default: 1): The number of Kafka consumer threads related to starting Timer EventsKAFKA_CONSUMER_SCHEDULED_TIMER_EVENTS_GROUP_ID
: Group of consumers related to starting timer eventsKAFKA_CONSUMER_STOP_SCHEDULED_TIMER_EVENTS_THREADS
- (default: 1): The number of Kafka consumer threads related to stopping Timer eventsKAFKA_CONSUMER_STOP_SCHEDULED_TIMER_EVENTS_GROUP_ID
: Group of consumers related to stopping timer eventsKAFKA_AUTH_EXCEPTION_RETRY_INTERVAL
: The interval between retries afterAuthorizationException
is thrown byKafkaConsumer
KAFKA_TOPIC_SCHEDULE_IN_SET
: Receives scheduled message setting requests from the Admin and Process engine microservicesKAFKA_TOPIC_SCHEDULER_IN_STOP
: Handles requests from the Admin and Process engine microservices to terminate scheduled messages.KAFKA_TOPIC_SCHEDULED_TIMER_EVENTS_IN_SET
: Needed to use Timer Events nodesKAFKA_TOPIC_SCHEDULED_TIMER_EVENTS_IN_STOP
: Needed to use Timer Events nodes
Each action available in the service corresponds to a Kafka event. A separate Kafka topic must be configured for each use-case.
Make sure the topics configured for this service don’t follow the engine pattern.
Configuring logging
The following environment variables could be set in order to control log levels:
LOGGING_LEVEL_ROOT: INFO
(Default): Root spring boot microservice logsLOGGING_LEVEL_APP
: App level logs
Was this page helpful?