Skip to main content
Data Search Service Overview
The Data Search service enables you to find specific data across process instances within your FlowX.AI platform. It uses Elasticsearch to perform fast, indexed searches across process data.

Prerequisites

Before using the Data Search service, ensure you have:
  • Elasticsearch cluster running and accessible
  • Indexing enabled in FlowX Engine configuration
  • Proper network connectivity between services
  • KAFKA_TOPIC_DATA_SEARCH_IN configured for requests
  • KAFKA_TOPIC_DATA_SEARCH_OUT configured for responses
  • Proper topic permissions and access
  • Process instances with indexed data
  • Searchable fields defined in Process Settings
  • Completed process instances for reliable results

Quick start

1

Set up your data process

Create a process that stores searchable data and configure field indexing.
2

Create a search process

Build a process with Kafka Send/Receive actions for search functionality.
3

Configure search parameters

Define your search criteria using simple or advanced syntax.
4

Handle search results

Process the returned data and display results to users.

1. Set up your data process

First, create a process that contains searchable data. Add a Service Task with a business rule:
output.put("application", {
  "client": {
    "personalInfo": {
      "firstName": "Sarah",
      "lastName": "Johnson", 
      "email": "sarah.johnson@techcorp.com",
      "phone": "+1-555-0101"
    },
    "address": {
      "country": "USA",
      "city": "San Francisco", 
      "state": "California",
      "zipCode": "94102"
    },
    "business": {
      "company": "TechCorp Inc",
      "industry": "Technology",
      "position": "CTO"
    },
    "account": {
      "type": "Premium",
      "status": "Active",
      "riskLevel": "Low"
    }
  }
});
Process Data Configuration
Critical Step: After creating your data structure, you must configure field indexing in Process Settings β†’ Data Search for the search to work.

Configure Field Indexing

Navigate to Process Settings β†’ Data Search and add the field paths you want to search:
Index Configuration
Fields that we indexed in our example:
  • application.client.personalInfo.firstName
  • application.client.personalInfo.lastName
  • application.client.personalInfo.email

2. Create your search process

Create a new process with a Send Message Task:
Search Process Configuration
Configuration:
  • Action Type: Kafka Send Action
  • Topic: KAFKA_TOPIC_DATA_SEARCH_IN

3. Configure search parameters

Choose your search approach based on your needs:

4. Handle search results

Add a Receive Message Task with:
  • Data Stream: KAFKA_TOPIC_DATA_SEARCH_OUT
Receive Search Results
Success response example:
Search Results
{
  "result": [
    {
      "processInstanceUUID": "b743ac99-4029-4b93-9ac3-4a45b4a62673",
      "state": "FINISHED",
      "processStartDate": "2025-05-28T12:18:50.532Z",
      "data": {
        "application": {
          "client": {
            "personalInfo": {
              "firstName": "Sarah",
              "lastName": "Johnson"
            }
          }
        }
      }
    }
  ],
  "tooManyResults": false,
  "success": true
}

Search parameters reference

Quick reference table

ParameterTypeMandatoryDescriptionExample
searchKeyStringYesSingle field path to search"application.client.personalInfo.lastName"
valueStringYesValue to search for (with searchKey)"Johnson"
searchKeysArrayYes (if multiple fields)Multiple field-value pairs (AND logic)[{"key": "field1", "value": "val1"}]
processDefinitionNamesArrayYesLimit to specific processes["client_onboarding"]
applicationIdsArrayNoSearch across applications["uuid-1", "uuid-2"]
statesArrayNoFilter by process states["FINISHED", "STARTED"]
processStartDateAfterStringNoInclude processes after date"2024-01-01T00:00:00Z"
processStartDateBeforeStringNoInclude processes before date"2024-12-31T23:59:59Z"
searchByPathsArrayNoSearch within array elements[{"key": "application.[].firstName", "value": "John"}]

Basic search parameters

Purpose: The field path to search in for single-field searches.Usage: Use dot notation to specify the exact path to the field you want to search or a specific key in the payload.Examples:
  • "application.client.personalInfo.firstName"
  • "application.client.address.city"
  • "application.client.business.industry"
Important: This field must be indexed in Process Settings β†’ Data Search to work.
Cannot be used together with searchKeys. Choose one approach.
Purpose: The exact value to search for when using searchKey.Usage: Must match the stored value exactly (case-sensitive).Examples:
  • "Johnson" - searches for exact lastName match
  • "Technology" - searches for exact industry match
  • "Active" - searches for exact status match
Note: Only used with searchKey, not with searchKeys.
Purpose: Array of key-value pairs for multi-field searches with AND logic.Usage: All conditions must match for a result to be returned.Format:
[
  {"key": "field.path.1", "value": "value1"},
  {"key": "field.path.2", "value": "value2"}
]
Example:
[
  {"key": "application.client.personalInfo.firstName", "value": "Sarah"},
  {"key": "application.client.business.industry", "value": "Technology"}
]
Cannot be used together with searchKey. Choose one approach.
Purpose: Search within array elements.Usage: Use [] notation to search within array elements. This searches through all items in the array.Example:
[
  {"key": "application.[].firstName", "value": "John"}
]

Filtering parameters

Purpose: Limit search to specific process definitions.Default: Searches all processes if omitted.Usage: Improves performance by narrowing search scope.Examples:
  • ["client_onboarding"] - search only in client processes
  • ["employee_registration", "contractor_onboarding"] - search in multiple process types
Best Practice: Always specify to improve search performance.
Purpose: Search across specific applications.Default: Searches current application if omitted.Usage: Enable cross-application searches.Examples:
  • ["8dd20844-2dc5-4445-83a5-bbbcc82bed5f"] - search in specific app
  • ["app-1-uuid", "app-2-uuid", "app-3-uuid"] - search across multiple apps
Use Case: Useful for consolidated reporting across different business units.
Purpose: Filter results by process instance status.Default: Returns all states if omitted.Available States:
  • "CREATED" - Process instance created but not started
  • "STARTED" - Process is currently running
  • "FINISHED" - Process completed successfully
  • "FAILED" - Process encountered an error and stopped
  • "TERMINATED" - Process was manually stopped/cancelled
  • "ONHOLD" - Process is paused or waiting for external input
Examples:
  • ["FINISHED"] - only completed processes
  • ["STARTED", "ONHOLD"] - active or paused processes
  • ["FAILED", "TERMINATED"] - processes that didn’t complete normally
Best Practice: Use ["FINISHED"] for most business searches to get complete data.

Date range parameters

Purpose: Include only processes started after the specified date.Format: ISO 8601 timestamp (YYYY-MM-DDTHH:MM:SSZ)Examples:
  • "2024-01-01T00:00:00Z" - processes started after Jan 1, 2024
  • "2024-06-15T09:30:00Z" - processes started after June 15, 2024 at 9:30 AM
Use Cases:
  • Monthly reports: "2024-03-01T00:00:00Z"
  • Recent activity: "2024-05-20T00:00:00Z"
Purpose: Include only processes started before the specified date.Format: ISO 8601 timestamp (YYYY-MM-DDTHH:MM:SSZ)Examples:
  • "2024-12-31T23:59:59Z" - processes started before end of 2024
  • "2024-06-30T23:59:59Z" - processes started before end of June 2024
Use Cases:
  • Historical analysis: "2024-01-01T00:00:00Z"
  • Quarterly reports: "2024-03-31T23:59:59Z"

Process states explained

Understanding process states is crucial for effective searching:
StateDescriptionWhen to Use
CREATEDProcess instance exists but hasn’t started executionRarely used for business searches
STARTEDProcess is actively runningFind ongoing processes, current workload
FINISHEDProcess completed successfullyMost common for business data searches
FAILEDProcess encountered an errorError analysis, troubleshooting
TERMINATEDProcess was manually cancelledAudit trails, cancelled applications
ONHOLDProcess is paused/waitingActive cases needing attention
Recommendation: Use ["FINISHED"] for most business searches to ensure you’re getting complete, reliable data.

Response structure

The search returns a JSON object with these fields:
  • result (Array): List of matching process instances, automatically sorted by processStartDate in descending order (newest first)
    • processInstanceUUID: Unique process identifier
    • state: Current process state
    • processStartDate: When the process started
    • data: The actual process data with your searchable fields
  • tooManyResults (Boolean): True if more than 50 results found (limit applied)
  • success (Boolean): Whether the search completed successfully
  • errorMessage (String): Error details if search failed

Use cases & examples

HR employee lookup

Search for employees by department, position, or location across HR systems.

Compliance Auditing

Locate specific transactions or approvals for regulatory compliance.

Business Intelligence

Analyze process data patterns and generate reports.
1

Customer calls with issue

Support agent needs to find customer’s account quickly.
2

Search by multiple criteria

{
  "searchKeys": [
    {"key": "customer.email", "value": "customer@email.com"},
    {"key": "customer.status", "value": "Active"}
  ],
  "processDefinitionNames": ["customer_onboarding"],
  "states": ["FINISHED"]
}
3

Get comprehensive results

Receive full customer profile with account details, order history, and support tickets.
4

Resolve customer issue

Use the retrieved data to address the customer’s concern effectively.

Best practices

  • Limit search scope using processDefinitionNames and states
  • Use date ranges for time-sensitive searches
  • Index only frequently searched fields to reduce storage overhead
  • Monitor search response times and optimize queries
  • Avoid wildcard searches on large datasets
  • Use consistent field naming across processes
  • Normalize data formats (dates, phone numbers, etc.)
  • Consider search patterns when designing data structures
  • Document indexed fields for team reference
  • Plan for data growth and scaling needs
  • Don’t index sensitive data (SSN, passwords, etc.)
  • Implement proper access controls for search endpoints
  • Log search activities for audit trails
  • Sanitize search inputs to prevent injection attacks
  • Follow data retention policies for search results
  • Handle empty results gracefully in your UI
  • Implement retry logic for failed searches
  • Provide meaningful error messages to users
  • Set reasonable timeouts for search operations
  • Monitor and alert on search failures

Troubleshooting

Possible Causes:
  • Fields not indexed in Process Settings β†’ Data Search
  • Incorrect field path (case-sensitive)
  • Process instances not in expected state
  • Elasticsearch indexing delay
Solutions:
  • Verify field indexing configuration
  • Check exact field paths in process data
  • Ensure processes are in FINISHED state
  • Wait 30-60 seconds after process completion
Possible Causes:
  • Query too broad (searching all data)
  • Elasticsearch cluster performance issues
  • Large dataset without proper filtering
Solutions:
  • Add more specific filters (processDefinitionNames, states)
  • Use date ranges to limit scope
  • Check Elasticsearch cluster health
  • Optimize indexing strategy
Possible Causes:
  • Typos in field paths
  • Field doesn’t exist in process data
  • Incorrect JSON format in request
Solutions:
  • Verify field paths exist in process instances
  • Check for case sensitivity
  • Validate JSON syntax
  • Test with simple field paths first
Possible Causes:
  • Kafka topics not properly configured
  • Network connectivity problems
  • Service authentication issues
Solutions:
  • Verify Kafka topic configuration
  • Check network connectivity
  • Validate service credentials
  • Review FlowX Engine logs

Configuration templates

{
  "searchKey": "your.field.path",
  "value": "search_value",
  "processDefinitionNames": ["your_process"],
  "states": ["FINISHED"]
}

⌘I