FlowX Data Search setup

The FlowX Data Search service enables powerful searching capabilities across your FlowX platform. This guide provides detailed instructions for setting up, configuring, and deploying the service in your environment.

Infrastructure prerequisites

The FlowX Data Search service requires the following infrastructure components:

Component	Purpose
Redis	Caching search results and configurations
Kafka	Message-based communication with the engine
Elasticsearch	Indexing and searching data

Configuration

Kafka configuration

Configure Kafka communication using these environment variables and properties:

Basic Kafka settings

Variable	Description	Default/Example
`SPRING_KAFKA_BOOTSTRAPSERVERS`	Address of Kafka server(s)	`localhost:9092`
`SPRING_KAFKA_SECURITY_PROTOCOL`	Security protocol for Kafka	`PLAINTEXT`
`KAFKA_CONSUMER_THREADS`	Number of Kafka consumer threads	`1`
`KAFKA_MESSAGE_MAX_BYTES`	Maximum message size	`52428800` (50MB)
`KAFKA_OAUTH_CLIENT_ID`	OAuth client ID for Kafka authentication	`kafka`
`KAFKA_OAUTH_CLIENT_SECRET`	OAuth client secret	`kafka-secret`
`KAFKA_OAUTH_TOKEN_ENDPOINT_URI`	OAuth token endpoint	`kafka.auth.localhost`

Topic naming configuration

The Data Search service uses a structured topic naming convention:

{package}.{environment}.{component}.{action}.{version}

For example: ai.flowx.dev.core.trigger.search.data.v1

Variable	Description	Default
`KAFKA_TOPIC_NAMING_SEPARATOR`	Primary separator for topic naming	`.`
`KAFKA_TOPIC_NAMING_SEPARATOR2`	Secondary separator	`-`
`KAFKA_TOPIC_NAMING_PACKAGE`	Package prefix	`ai.flowx.`
`KAFKA_TOPIC_NAMING_ENVIRONMENT`	Environment name	`dev.`
`KAFKA_TOPIC_NAMING_VERSION`	Version suffix	`.v1`

Kafka topics

The service uses these specific topics:

Topic	Default Value	Purpose
`KAFKA_TOPIC_DATA_SEARCH_IN`	`ai.flowx.dev.core.trigger.search.data.v1`	Incoming search requests
`KAFKA_TOPIC_DATA_SEARCH_OUT`	`ai.flowx.dev.engine.receive.core.search.data.results.v1`	Outgoing search results

Elasticsearch configuration

Configure Elasticsearch connection using the following environment variables:

Variable	Description	Default Value	Example
`SPRING_ELASTICSEARCH_REST_URIS`	URL(s) of Elasticsearch nodes (no protocol)	-	`elasticsearch:9200`
`SPRING_ELASTICSEARCH_REST_PROTOCOL`	Connection protocol	`https`	`https` or `http`
`SPRING_ELASTICSEARCH_REST_DISABLESSL`	Disable SSL verification	`false`	`false`
`SPRING_ELASTICSEARCH_REST_USERNAME`	Authentication username	-	`elastic`
`SPRING_ELASTICSEARCH_REST_PASSWORD`	Authentication password	-	`your-password`
`SPRING_ELASTICSEARCH_INDEXSETTINGS_NAME`	Index name for search data	`process_instance`	`process_instance`

Security configuration

Configure authentication and authorization with these variables:

Variable	Description	Example
`SECURITY_OAUTH2_BASESERVERURL`	Base URL for OAuth2 server	`https://keycloak.example.com/auth`
`SECURITY_OAUTH2_CLIENT_CLIENTID`	OAuth2 client ID	`data-search-service`
`SECURITY_OAUTH2_CLIENT_CLIENTSECRET`	OAuth2 client secret	`data-search-service-secret`
`SECURITY_OAUTH2_REALM`	OAuth2 realm name	`flowx`

Logging configuration

Control the verbosity of logs with these variables:

Variable	Description	Default	Example
`LOGGING_LEVEL_ROOT`	Root Spring Boot log level	`INFO`	`ERROR`
`LOGGING_LEVEL_APP`	Application-specific log level	`INFO`	`DEBUG`

Elasticsearch index configuration

The Data Search service creates and manages Elasticsearch indices based on the configured index pattern. The default index name is process_instance.

Index pattern

The service derives the index pattern from the spring.elasticsearch.index-settings.name property. This pattern is used to query across multiple indices that match the pattern.

Sample search query

Below is an example of a search query generated by the Data Search service for Elasticsearch:

{
  "query": {
    "bool": {
      "adjust_pure_negative": true,
      "boost": 1,
      "must": [
        {
          "nested": {
            "boost": 1,
            "ignore_unmapped": false,
            "path": "keyIdentifiers",
            "query": {
              "bool": {
                "adjust_pure_negative": true,
                "boost": 1,
                "must": [
                  {
                    "match": {
                      "keyIdentifiers.key.keyword": {
                        "query": "astonishingAttribute",
                        "operator": "OR"
                      }
                    }
                  },
                  {
                    "match": {
                      "keyIdentifiers.originalValue.keyword": {
                        "query": "OriginalGangsta",
                        "operator": "OR"
                      }
                    }
                  }
                ]
              }
            },
            "score_mode": "none"
          }
        },
        {
          "terms": {
            "boost": 1,
            "processDefinitionName.keyword": [
              "TEST_PORCESS_NAME_0",
              "TEST_PORCESS_NAME_1"
            ]
          }
        }
      ]
    }
  }
}

Troubleshooting

Common issues

Elasticsearch connection problems:
- Verify Elasticsearch is running and accessible
- Check if credentials are correct
- Ensure SSL settings match your environment
Kafka Communication Issues:
- Verify Kafka topics exist and are properly configured
- Check Kafka permissions for the service
- Ensure bootstrap servers are correctly specified
Search Not Returning Results:
- Verify index pattern matches existing indices
- Check if data is being properly indexed
- Review search query format for errors

Logs analysis

Monitor logs for errors and warnings:

# For Docker
docker logs flowx-data-search

# For Kubernetes
kubectl logs deployment/data-search

Integration with Kibana

Kibana provides a powerful interface for visualizing and exploring data indexed by the Data Search service.

Using Kibana with FlowX Data Search

Connect Kibana to the same Elasticsearch instance
Create an index pattern matching your configured index name
Use the Discover tab to explore indexed data
Create visualizations and dashboards based on your data

Kibana is an open-source data visualization and exploration tool designed primarily for Elasticsearch. It serves as the visualization layer for the Elastic Stack, allowing users to interact with their data stored in Elasticsearch to perform various activities such as querying, analyzing, and visualizing data. For more information, visit the Kibana official documentation.

Best practices

Security:
- Store sensitive credentials in Kubernetes Secrets
- Use TLS for Elasticsearch and Kafka communication
- Implement network policies to restrict access
Performance:
- Scale the number of replicas based on query load
- Adjust Kafka consumer threads based on message volume
- Configure appropriate resource limits and requests
Monitoring:
- Set up monitoring for Elasticsearch, Kafka, and Redis
- Create alerts for service availability and performance
- Monitor disk space for Elasticsearch data nodes

Microservices

Plugins

Observability

Microservices access rights

Plugins access rights

FlowX Data Search setup

Infrastructure prerequisites

Configuration

Kafka configuration

Basic Kafka settings

Topic naming configuration

Kafka topics

Elasticsearch configuration

Security configuration

Logging configuration

Elasticsearch index configuration

Index pattern

Sample search query

Troubleshooting

Common issues

Logs analysis

Integration with Kibana

Using Kibana with FlowX Data Search

Best practices

​Infrastructure prerequisites

​Configuration

​Kafka configuration

​Basic Kafka settings

​Topic naming configuration

​Kafka topics

​Elasticsearch configuration

​Security configuration

​Logging configuration

​Elasticsearch index configuration

​Index pattern

​Sample search query

​Troubleshooting

​Common issues

​Logs analysis

​Integration with Kibana

​Using Kibana with FlowX Data Search

​Best practices

Infrastructure prerequisites

Configuration

Kafka configuration

Basic Kafka settings

Topic naming configuration

Kafka topics

Elasticsearch configuration

Security configuration

Logging configuration

Elasticsearch index configuration

Index pattern

Sample search query

Troubleshooting

Common issues

Logs analysis

Integration with Kibana

Using Kibana with FlowX Data Search

Best practices