This guide shows you how to enable process instance indexing using either Kafka or HTTP transport strategies.
Strategy | Best for | Pros | Cons |
---|---|---|---|
Kafka (recommended) | Production environments with high throughput | Fire-and-forget communication, time-based partitioning, better performance, decoupled architecture | Requires Kafka Connect setup |
HTTP | Development or simple setups | Direct connection, easier setup | Blocking operations, no time-based partitioning, tighter coupling |
transforms.routeTS.timestamp.format
configuration (see later in this guide). The HTTP strategy does not support time-based partitioning as a built-in feature.FLOWX_INDEXING_ENABLED
defaults to true. Only set this variable if you want to disable indexing by setting it to false.yyyyMM
format for time-based partitioning (Kafka only)yyyyww
)transforms.routeTS.timestamp.format
: Controls index partitioning (monthly=yyyyMM
, daily=yyyyMMdd
)transforms.routeTS.topic.format
: Must start with your configured index namebatch.size
: Adjust based on throughput needs (1000 is good default)Environment Variable | Default Value |
---|---|
KAFKA_TOPIC_PROCESS_INDEX_OUT | ai.flowx.dev.core.index.process.v1 |
Database Partitioning | Elasticsearch Format | Index Pattern | Best For |
---|---|---|---|
Monthly | yyyyMM | process_instance-202406 | Medium volume |
Weekly | yyyyww | process_instance-202426 | High volume |
Daily | yyyyMMdd | process_instance-20240615 | Very high volume |
FLOWX_INDEXING_ENABLED=false
)batch.size
in connector config