FlowX Data Search setup
This guide will walk you through the process of setting up the Data Search service using a Docker image.
Infrastructure prerequisites
Before proceeding, ensure the following components are set up:
- Redis - version 6.0 or higher
- Kafka - version 2.8 or higher
- Elasticsearch - version 7.11.0 or higher
Dependencies
- Kafka: Used for communication with the engine
- Elasticsearch: Used for indexing and searching data
- Redis: Used for caching
Configuration
Kafka Configuration
Set the following Kafka-related configurations using environment variables:
SPRING_KAFKA_BOOTSTRAP_SERVERS
: Address of the Kafka serverKAFKA_TOPIC_DATA_SEARCH_IN
: The Kafka topic for the search service requests to the engineKAFKA_TOPIC_DATA_SEARCH_OUT
: Where the engine awaits for the responseKAFKA_CONSUMER_THREADS
: Number of Kafka consumer threads
Elasticsearch configuration
Set the following Elasticsearch-related configurations using environment variables:
SPRING_ELASTICSEARCH_REST_URIS
SPRING_ELASTICSEARCH_REST_PROTOCOL
SPRING_ELASTICSEARCH_REST_DISABLESSL
SPRING_ELASTICSEARCH_REST_USERNAME
SPRING_ELASTICSEARCH_REST_PASSWORD
SPRING_ELASTICSEARCH_INDEX_SETTINGS_NAME
(default:process_instance
)
spring:
elasticsearch:
rest:
protocol: https
uris: localhost:9200
disableSsl: false
username: ""
password: ""
index-settings:
name: process_instance
Authorization & Access Roles Configuration
Set the following environment variables to connect to the identity management platform:
SECURITY_OAUTH2_BASE_SERVER_URL
SECURITY_OAUTH2_CLIENT_CLIENT_ID
SECURITY_OAUTH2_REALM
Logging Configuration
Control log levels using these environment variables:
LOGGING_LEVEL_ROOT
: For root spring boot microservice logsLOGGING_LEVEL_APP
: For app-level logs
Elasticsearch
Data search in Elasticsearch operates against an index pattern representing multiple indices. The index pattern is derived from the configuration property spring.elasticsearch.index-settings.name
.
Here’s an example filter for use in Kibana (generated by data search):
{
"query": {
"bool": {
"adjust_pure_negative": true,
"boost": 1,
"must": [
{
"nested": {
"boost": 1,
"ignore_unmapped": false,
"path": "keyIdentifiers",
"query": {
"bool": {
"adjust_pure_negative": true,
"boost": 1,
"must": [
{
"match": {
"keyIdentifiers.key.keyword": {
"auto_generate_synonyms_phrase_query": true,
"boost": 1,
"fuzzy_transpositions": true,
"lenient": false,
"max_expansions": 50,
"operator": "OR",
"prefix_length": 0,
"query": "astonishingAttribute",
"zero_terms_query": "NONE"
}
}
},
{
"match": {
"keyIdentifiers.originalValue.keyword": {
"auto_generate_synonyms_phrase_query": true,
"boost": 1,
"fuzzy_transpositions": true,
"lenient": false,
"max_expansions": 50,
"operator": "OR",
"prefix_length": 0,
"query": "OriginalGangsta",
"zero_terms_query": "NONE"
}
}
}
]
}
},
"score_mode": "none"
}
},
{
"terms": {
"boost": 1,
"processDefinitionName.keyword": [
"TEST_PORCESS_NAME_0",
"TEST_PORCESS_NAME_1"
]
}
}
]
}
}
}
Kibana is an open-source data visualization and exploration tool designed primarily for Elasticsearch. It serves as the visualization layer for the Elastic Stack, allowing users to interact with their data stored in Elasticsearch to perform various activities such as querying, analyzing, and visualizing data.
For more information about Kibana and its capabilities, visit the Kibana official documentation. This resource provides in-depth guidance, tutorials, and documentation on how to use Kibana effectively for data visualization, analysis, and dashboard creation.
Was this page helpful?