Configuring Elasticsearch indexing
This section provides configuration steps for enabling process instance indexing using the Kafka transport strategy.
Before proceeding, it is recommended to familiarize yourself with Elasticsearch and its indexing process by referring to the Intro to Elasticsearch section.
Intro to Elasticsearch
Configuration updates
To enable Kafka indexing strategy, the previous configuration parameter flowx.use-elasticsearch
is being replaced. However, to ensure backward compatibility, it will still be preserved in the configuration. Below is an example of how to configure it:
Instead of the removed configuration, a new configuration area has been added:
The flowx.indexing.enabled
property determines whether indexing with Elasticsearch is enabled. When set to false or missing, no indexing will be performed for any entities defined below. When set to true, indexing with Elasticsearch is enabled.
If the FlowX indexing configuration is set to false, the following configuration information and guidelines are not applicable to your use case.
The flowx.indexing.processInstance.indexing-type
property defines the indexing strategy for process instances. It can have one of the following values:
- no-indexing: No indexing will be performed for process instances.
- http: Direct connection from the process engine to Elasticsearch through HTTP calls.
- kafka: Data will be sent to be indexed via a Kafka topic using the new strategy. To implement this strategy, the Kafka Connect with Elasticsearch Sink Connector must be deployed in the infrastructure.
Configuration steps
To enable indexing with Elasticsearch for the entire application, update the process-engine configuration with the following parameters:
FLOWX_INDEXING_ENABLED
: Set this parameter totrue
to enable indexing with Elastisearch for the entire application.
Variable Name | Enabled | Description |
---|---|---|
FLOWX_INDEXING_ENABLED | true | Indexing with Elasticsearch for the whole app is enabled |
FLOWX_INDEXING_ENABLED | false | Indexing with Elasticsearch for the whole app is disabled |
FLOWX_INDEXING_PROCESSINSTANCE_INDEXING_TYPE
: Set this parameter tokafka
to use the Kafka transport strategy for indexing process instances.
Variable Name | Indexing Type - Values | Definition |
---|---|---|
FLOWX_INDEXING_PROCESSINSTANCE_INDEXING_TYPE | no-indexing | No indexing is performed for process instances |
FLOWX_INDEXING_PROCESSINSTANCE_INDEXING_TYPE | http | Process instances are indexed via HTTP (direct connection from process-engine to Elasticsearch thorugh HTTP calls) |
FLOWX_INDEXING_PROCESSINSTANCE_INDEXING_TYPE | kafka | Process instances are indexed via Kafka (send data to be indexed through a kafka topic - the new strategy for the applied solution) |
FLOWX_INDEXING_PROCESSINSTANCE_INDEX_NAME
: Specify the name of the index used for process instances.
Variable Name | Values | Definition |
---|---|---|
FLOWX_INDEXING_PROCESSINSTANCE_INDEXING_INDEX_NAME | process_instance | The name of the index used for storing process instances. It is also part of the search pattern |
FLOWX_INDEXING_PROCESSINSTANCE_SHARDS
: Set the number of shards for the index.
Variable Name | Values | Definition |
---|---|---|
FLOWX_INDEXING_PROCESSINSTANCE_SHARDS | 1 | The number of shards for the Elasticsearch index storing process instances |
FLOWX_INDEXING_PROCESSINSTANCE_REPLICAS
: Set the number of replicas for the index.
Variable Name | Values | Definition |
---|---|---|
FLOWX_INDEXING_PROCESSINSTANCE_REPLICAS | 1 | The number of replicas for the Elasticsearch index storing process instances |
For Kafka indexing, the Kafka Connect with Elasticsearch Sink Connector must be deployed in the infrastructure.
Elasticsearch Service Sink Connector
Configuration examples
Kafka Connect
- Assumes kafka cluster installed with strimzi operator and Elasticsearch with eck-operator
- Can save the image built by kafka connect to a local registry and comment build section
Kafka Elasticsearch Connector
HTTP indexing
If you don’t want to remove the existing configuration parameters, you can use the following example:
Querying Elasticsearch
To read from multiple indices, queries in Elasticsearch have been updated. The queries now run against an index pattern that identifies multiple indices instead of a single index. The index pattern is derived from the value defined in the configuration property:
flowx.indexing.processInstance.index-name
Kafka topics - process events messages
This topic is used for sending the data to be indexed from Process engine. The data from this topic will be read by Kafka Connect.
- Key:
${kafka.topic.process.index.out}
- Value:
${kafka.topic.naming.prefix}.core.index.process${kafka.topic.naming.suffix}
Default parameter (env var) | Default FLOWX.AI value (can be overwritten) |
---|---|
KAFKA_TOPIC_PROCESS_INDEX_OUT | ai.flowx.dev.core.index.process.v1 |
The topic name, defined in the value, will be used by Kafka Connect as source for the messages to be sent to Elasticsearch for indexing.
The attribute indexLastUpdatedTime
is new and will be populated for the kafka-connect strategy. This will tell the timestamp when the last operation was done on the object in the index.ß
Elasticsearch update (index template)
The mappings between messages and Elasticsearch data types need to be specified. This is achieved through an index template created by the process engine during startup. The template applies to indices starting with the value defined in flowx.indexing.processInstance.index-name
config. Here’s an example of the index template:
Here are some guidelines to help you get started:
Configuration guidelines
Was this page helpful?