FlowX Data Search setup

​

Overview

The FlowX Data Search service enables powerful searching capabilities across your FlowX platform. This guide provides detailed instructions for setting up, configuring, and deploying the service in your environment.

​

Quick start

# 1. Ensure infrastructure prerequisites are met (Redis, Kafka, Elasticsearch)
# 2. Configure your environment variables in a data-search.yaml file
# 3. Deploy the service
kubectl apply -f data-search.yaml
# 4. Verify the deployment
kubectl get deployment data-search
kubectl logs deployment/data-search

​

Infrastructure prerequisites

The FlowX Data Search service requires the following infrastructure components:

​

Configuration

​

Kafka configuration

Configure Kafka communication using these environment variables and properties:

​

Basic Kafka settings

​

Topic naming configuration

The Data Search service uses a structured topic naming convention:

{package}.{environment}.{component}.{action}.{version}

For example: ai.flowx.dev.core.trigger.search.data.v1

​

Kafka topics

The service uses these specific topics:

​

Elasticsearch configuration

Set up Elasticsearch connectivity with these environment variables:

​

Security configuration

Configure authentication and authorization with these variables:

​

Logging configuration

Control the verbosity of logs with these variables:

​

Elasticsearch configuration

Set up Elasticsearch connectivity with these environment variables:

​

Elasticsearch index configuration

The Data Search service creates and manages Elasticsearch indices based on the configured index pattern. The default index name is process_instance.

​

Index pattern

The service derives the index pattern from the spring.elasticsearch.index-settings.name property. This pattern is used to query across multiple indices that match the pattern.

​

Sample search query

Below is an example of a search query generated by the Data Search service for Elasticsearch:

{
  "query": {
    "bool": {
      "adjust_pure_negative": true,
      "boost": 1,
      "must": [
        {
          "nested": {
            "boost": 1,
            "ignore_unmapped": false,
            "path": "keyIdentifiers",
            "query": {
              "bool": {
                "adjust_pure_negative": true,
                "boost": 1,
                "must": [
                  {
                    "match": {
                      "keyIdentifiers.key.keyword": {
                        "query": "astonishingAttribute",
                        "operator": "OR"
                      }
                    }
                  },
                  {
                    "match": {
                      "keyIdentifiers.originalValue.keyword": {
                        "query": "OriginalGangsta",
                        "operator": "OR"
                      }
                    }
                  }
                ]
              }
            },
            "score_mode": "none"
          }
        },
        {
          "terms": {
            "boost": 1,
            "processDefinitionName.keyword": [
              "TEST_PORCESS_NAME_0",
              "TEST_PORCESS_NAME_1"
            ]
          }
        }
      ]
    }
  }
}

​

Troubleshooting

​

Common issues

1. Elasticsearch connection problems:

   • Verify Elasticsearch is running and accessible
   • Check if credentials are correct
   • Ensure SSL settings match your environment

2. Kafka Communication Issues:

   • Verify Kafka topics exist and are properly configured
   • Check Kafka permissions for the service
   • Ensure bootstrap servers are correctly specified

3. Search Not Returning Results:

   • Verify index pattern matches existing indices
   • Check if data is being properly indexed
   • Review search query format for errors

​

Logs analysis

Monitor logs for errors and warnings:

# For Docker
docker logs flowx-data-search

# For Kubernetes
kubectl logs deployment/data-search

​

Integration with Kibana

Kibana provides a powerful interface for visualizing and exploring data indexed by the Data Search service.

​

Using Kibana with FlowX Data Search

1. Connect Kibana to the same Elasticsearch instance
2. Create an index pattern matching your configured index name
3. Use the Discover tab to explore indexed data
4. Create visualizations and dashboards based on your data

​

Best practices

1. Security:

   • Store sensitive credentials in Kubernetes Secrets
   • Use TLS for Elasticsearch and Kafka communication
   • Implement network policies to restrict access

2. Performance:

   • Scale the number of replicas based on query load
   • Adjust Kafka consumer threads based on message volume
   • Configure appropriate resource limits and requests

3. Monitoring:

   • Set up monitoring for Elasticsearch, Kafka, and Redis
   • Create alerts for service availability and performance
   • Monitor disk space for Elasticsearch data nodes