Knowledge Graph Setup

Overview

The Knowledge Graph (KG) service is a foundational microservice in FlowX.AI’s AI Agent ecosystem that provides a distributed graph database solution for managing AI Agent state, enabling Retrieval-Augmented Generation (RAG), and facilitating multi-agent collaboration.

The Knowledge Graph service uses DGraph as the underlying graph database technology and is essential for AI Agent operations, conversation state management, and cross-agent data sharing.

Key Capabilities

AI Agent State Management

Persistent storage and retrieval of AI Agent states across horizontal scaling scenarios

RAG Support

Enhanced LLM prompts with domain-specific data through efficient graph-based retrieval

Multi-Agent Collaboration

Transparent agent integration with shared context and state management

Connected Data

Property graph implementation for scalable, queryable data relationships

Infrastructure Prerequisites

Before setting up the Knowledge Graph service, ensure the following components are installed and configured:

Core Requirements

Component	Version	Purpose
Docker Engine	17.06+	Container runtime
DGraph	Latest	Graph database backend
Kafka	2.8+	Event-driven communication
Redis	6.0+	Caching layer
PostgreSQL	13+	Metadata storage (Designer phase)

DGraph Cluster Requirements

For production deployments:

Minimum 3 Alpha nodes - Store graph data and handle queries
Minimum 3 Zero nodes - Manage cluster state and coordination
Persistent storage - For data durability
Network connectivity - Between all nodes in the cluster

FlowX Dependencies

Process Engine - Must be deployed and running
Advancing Controller - Required for workflow integration
Existing Kafka instance - Used by other FlowX services
Redis instance - Shared with other FlowX microservices

Configuration Parameters

Core Service Configuration

Core Service Settings

# Knowledge Graph Service
KG_SERVICE_NAME=knowledge-graph
KG_SERVICE_PORT=8080
KG_SERVICE_HOST=0.0.0.0
KG_HEALTH_CHECK_INTERVAL=30s

# Service Discovery
SPRING_APPLICATION_NAME=knowledge-graph
SERVER_PORT=8080

DGraph Cluster Configuration

DGraph Settings

# DGraph Endpoints
DGRAPH_ALPHA_ENDPOINT=http://dgraph-alpha:8080
DGRAPH_ADMIN_ENDPOINT=http://dgraph-alpha:8080/admin
DGRAPH_ZERO_ENDPOINT=http://dgraph-zero:5080

# Cluster Configuration
DGRAPH_ALPHA_REPLICAS=3
DGRAPH_ZERO_REPLICAS=3
DGRAPH_STORAGE_PATH=/dgraph/data

# Performance Tuning
DGRAPH_MEMORY_MB=8192
DGRAPH_CACHE_SIZE_MB=2048
DGRAPH_MAX_CONNECTIONS=100
DGRAPH_QUERY_TIMEOUT=300s

Integration Configuration

FlowX Integration

# Kafka Configuration
KAFKA_BOOTSTRAP_SERVERS=${KAFKA_BOOTSTRAP_SERVERS}
KAFKA_CONSUMER_GROUP_ID=kg-service-group
KAFKA_AUTO_OFFSET_RESET=earliest

# Kafka Topics
KAFKA_TOPICS_AI_AGENT_STATE=ai-agent-state
KAFKA_TOPICS_MULTI_AGENT_COLLAB=multi-agent-collaboration
KAFKA_TOPICS_RAG_QUERIES=rag-queries

# FlowX Service Integration
FLOWX_ENGINE_ENDPOINT=http://process-engine:8080
FLOWX_ADVANCING_CONTROLLER_ENDPOINT=http://advancing-controller:8080

# Redis Configuration
REDIS_HOST=${REDIS_HOST}
REDIS_PORT=${REDIS_PORT}
REDIS_PASSWORD=${REDIS_PASSWORD}
REDIS_DATABASE=5

Security Configuration

Basic Authentication
Enterprise Security

Basic Auth (DGraph OSS)

# Shared token authentication
DGRAPH_AUTH_TOKEN=${DGRAPH_AUTH_TOKEN}
DGRAPH_AUTH_ENABLED=true

# TLS Configuration
DGRAPH_TLS_ENABLED=false
DGRAPH_TLS_CERT_PATH=/etc/ssl/certs/dgraph.crt
DGRAPH_TLS_KEY_PATH=/etc/ssl/private/dgraph.key

Deployment

Docker Compose Deployment

version: '3.8'

services:
  # DGraph Zero Nodes (Cluster Management)
  dgraph-zero-1:
    image: dgraph/dgraph:latest
    command: |
      dgraph zero 
      --my=dgraph-zero-1:5080 
      --replicas=3 
      --raft="idx=1"
    ports:
      - "5080:5080"
    volumes:
      - dgraph-zero-1:/dgraph
    networks:
      - dgraph-network

  dgraph-zero-2:
    image: dgraph/dgraph:latest
    command: |
      dgraph zero 
      --my=dgraph-zero-2:5080 
      --replicas=3 
      --raft="idx=2" 
      --peer=dgraph-zero-1:5080
    ports:
      - "5081:5080"
    volumes:
      - dgraph-zero-2:/dgraph
    networks:
      - dgraph-network
    depends_on:
      - dgraph-zero-1

  dgraph-zero-3:
    image: dgraph/dgraph:latest
    command: |
      dgraph zero 
      --my=dgraph-zero-3:5080 
      --replicas=3 
      --raft="idx=3" 
      --peer=dgraph-zero-1:5080
    ports:
      - "5082:5080"
    volumes:
      - dgraph-zero-3:/dgraph
    networks:
      - dgraph-network
    depends_on:
      - dgraph-zero-1

  # DGraph Alpha Nodes (Data Storage)
  dgraph-alpha-1:
    image: dgraph/dgraph:latest
    command: |
      dgraph alpha 
      --my=dgraph-alpha-1:7080 
      --zero=dgraph-zero-1:5080,dgraph-zero-2:5080,dgraph-zero-3:5080
      --security="whitelist=0.0.0.0/0"
    ports:
      - "8080:8080"
      - "9080:9080"
    volumes:
      - dgraph-alpha-1:/dgraph
    networks:
      - dgraph-network
    depends_on:
      - dgraph-zero-1
      - dgraph-zero-2
      - dgraph-zero-3

  dgraph-alpha-2:
    image: dgraph/dgraph:latest
    command: |
      dgraph alpha 
      --my=dgraph-alpha-2:7080 
      --zero=dgraph-zero-1:5080,dgraph-zero-2:5080,dgraph-zero-3:5080
      --security="whitelist=0.0.0.0/0"
    ports:
      - "8081:8080"
      - "9081:9080"
    volumes:
      - dgraph-alpha-2:/dgraph
    networks:
      - dgraph-network
    depends_on:
      - dgraph-zero-1
      - dgraph-zero-2
      - dgraph-zero-3

  dgraph-alpha-3:
    image: dgraph/dgraph:latest
    command: |
      dgraph alpha 
      --my=dgraph-alpha-3:7080 
      --zero=dgraph-zero-1:5080,dgraph-zero-2:5080,dgraph-zero-3:5080
      --security="whitelist=0.0.0.0/0"
    ports:
      - "8082:8080"
      - "9082:9080"
    volumes:
      - dgraph-alpha-3:/dgraph
    networks:
      - dgraph-network
    depends_on:
      - dgraph-zero-1
      - dgraph-zero-2
      - dgraph-zero-3

  # Knowledge Graph Service
  knowledge-graph:
    image: flowx/knowledge-graph:latest
    ports:
      - "8090:8080"
    environment:
      - DGRAPH_ALPHA_ENDPOINT=http://dgraph-alpha-1:8080
      - KAFKA_BOOTSTRAP_SERVERS=kafka:9092
      - REDIS_HOST=redis
      - SPRING_PROFILES_ACTIVE=docker
    networks:
      - dgraph-network
      - flowx-network
    depends_on:
      - dgraph-alpha-1
      - dgraph-alpha-2
      - dgraph-alpha-3

volumes:
  dgraph-zero-1:
  dgraph-zero-2:
  dgraph-zero-3:
  dgraph-alpha-1:
  dgraph-alpha-2:
  dgraph-alpha-3:

networks:
  dgraph-network:
    driver: bridge
  flowx-network:
    external: true

Kubernetes Deployment

apiVersion: v1
kind: Namespace
metadata:
  name: flowx-kg
  labels:
    name: flowx-kg

Performance Benchmarks

Based on synthetic testing with realistic AI Agent workloads:

Read Performance

20-45ms average response time

Query complexity: Multi-hop graph traversals
Concurrent threads: 10-15
Data scale: 4M+ nodes, 8M+ relationships

Write Performance

10-20ms per node average

Includes relationship creation
Batch operations supported
ACID transaction guarantees

Test Data Scale

Synthetic Data Stats

{
  "conversations": 4156,
  "threads": 41560,
  "messages": 831000,
  "actions": 4154000,
  "outcomes": 4155000,
  "feedback": 4155000
}

Health Checks and Monitoring

Service Health Endpoints

Health Check URLs

# Knowledge Graph Service Health
curl http://localhost:8090/actuator/health

# DGraph Cluster Health
curl http://localhost:8080/health

# DGraph Cluster State
curl http://localhost:8080/state

# Performance Metrics
curl http://localhost:8090/actuator/metrics

Key Metrics to Monitor

DGraph Cluster Metrics

Alpha node health - All nodes responding
Zero node health - Cluster coordination status
Memory usage - RAM consumption per node
Disk usage - Storage utilization
Query latency - Response time percentiles

Knowledge Graph Service Metrics

API response times - HTTP request latency
Kafka consumer lag - Message processing delay
Redis cache hit rate - Caching effectiveness
Error rates - Failed requests percentage

Schema Management

The Knowledge Graph service automatically manages schemas for AI Agent operations:

Core Schema Types

Conversation Schema
AI Agent Schema
RAG Schema

Conversation Types

type Conversation {
  id: ID!
  tenantId: String! @index(exact)
  userId: String! @index(exact)
  createdAt: DateTime!
  updatedAt: DateTime!
  threads: [Thread!]! @hasInverse(field: conversation)
  status: ConversationStatus!
}

type Thread {
  id: ID!
  conversation: Conversation! @hasInverse(field: threads)
  messages: [Message!]! @hasInverse(field: thread)
  executionPlan: [Action!]! @hasInverse(field: thread)
  createdAt: DateTime!
}

type Message {
  id: ID!
  thread: Thread! @hasInverse(field: messages)
  content: String!
  role: MessageRole!
  timestamp: DateTime!
  metadata: String
}

Troubleshooting

Cluster Startup Issues

Problem: DGraph nodes fail to join clusterSolutions:

Check network connectivity between nodes
Verify Zero nodes start before Alpha nodes
Ensure persistent volumes are properly mounted
Check for port conflicts (5080, 8080, 9080)

Debug Commands

# Check node logs
docker logs dgraph-zero-1
docker logs dgraph-alpha-1

# Verify cluster membership
curl http://localhost:8080/state | jq '.groups'

Performance Issues

Problem: Slow query response timesSolutions:

Increase memory allocation for Alpha nodes
Add appropriate indexes for frequent queries
Optimize query patterns to reduce traversal depth
Consider adding more Alpha nodes for read scaling

Performance Tuning

# Check memory usage
curl http://localhost:8080/debug/jemalloc

# Monitor query performance
curl http://localhost:8080/debug/store

Data Consistency Issues

Problem: Inconsistent data across cluster nodesSolutions:

Verify all Zero nodes are healthy
Check for network partitions
Monitor Raft consensus logs
Perform cluster backup and restore if needed

Consistency Checks

# Check Raft status
curl http://localhost:5080/debug/raft

# Verify data consistency
curl -X POST localhost:8080/query -d '{ checkData(func: has(id)) { count(uid) } }'

Security Best Practices

For production deployments, implement these security measures:

Network Security
- Use private networks for inter-node communication
- Implement firewall rules for DGraph ports
- Enable TLS for all communications
Authentication & Authorization
- Use DGraph Enterprise ACLs for fine-grained access control
- Implement JWT-based authentication for API access
- Rotate authentication tokens regularly
Data Protection
- Enable encryption at rest (DGraph Enterprise)
- Implement backup encryption
- Use secure communication protocols
Monitoring & Auditing
- Enable audit logging (DGraph Enterprise)
- Monitor for suspicious query patterns
- Set up alerts for security events

Backup and Recovery

Automated Backup (DGraph Enterprise)

Backup Configuration

# Environment variables for backup
DGRAPH_BACKUP_DESTINATION=s3://your-backup-bucket
DGRAPH_BACKUP_ACCESS_KEY=${AWS_ACCESS_KEY}
DGRAPH_BACKUP_SECRET_KEY=${AWS_SECRET_KEY}
DGRAPH_BACKUP_SCHEDULE="0 2 * * *" # Daily at 2 AM

# Manual backup command
curl -X POST localhost:8080/admin/backup \
  -H "Content-Type: application/json" \
  -d '{"destination": "s3://your-backup-bucket/backup-$(date +%Y%m%d)"}'

Export/Import (DGraph OSS)

Manual Export/Import

# Export data
curl -X POST localhost:8080/admin/export

# Import data (during cluster initialization)
dgraph bulk -r /path/to/export -s /path/to/schema.graphql

Developer Guidelines

Schema Contribution Standards

The Knowledge Graph service acts as “Database as a Service” for all FlowX microservices. Follow these guidelines when contributing schemas:

Naming Conventions

Type Naming Rules:

Use PascalCase for all type definitions
Prefix types with the microservice name to avoid conflicts
Follow the pattern: {MicroserviceName}{ActualDataType}

Examples

# ✅ Correct naming
type ModelsAgentModelConfiguration {
  id: ID!
  name: String!
}

type PlannerExecutionStep {
  id: ID!
  action: String!
}

# ❌ Incorrect naming  
type AgentModelConfiguration {  # Missing microservice prefix
  id: ID!
}

Schema Organization

File Structure:

/api
  /graphql
    /kag
      /common.graphql          # General purpose value objects
      /services
        /{microservice_id}     # e.g., planner, models, chat
          /{schema_name}.graphql # e.g., agent_config.graphql

Organization Rules:

Place schemas in the most specific microservice folder
Use descriptive file names that reflect the domain
Keep common types in /common.graphql only if truly general-purpose

Indexing Best Practices

Add indexes only for properties that are frequently queried:

Indexing Examples

type AiEmbedding {
  id: ID!
  content: String! @search(by: [fulltext])
  tenantId: String! @index(exact)
  embeddings: [Float!]! @embedding @search(by: ["hnsw(metric: cosine)"])
  createdAt: DateTime! @index(day)
}

Index Types:

@index(exact) - For exact matches (IDs, enum values)
@search(by: [fulltext]) - For text search
@embedding @search(by: ["hnsw"]) - For vector similarity
@index(day) - For date-based queries

Breaking Changes Prevention

❌ Avoid These Breaking Changes:

Removing existing fields from types
Changing field types (String to Int)
Removing types that are in use
Changing required fields to optional or vice versa

✅ Safe Changes:

Adding new optional fields
Adding new types
Adding new indexes
Extending enums with new values

A schema registry will be implemented to automatically detect breaking changes in the future.

Schema Deployment Process

Local Development
Production Deployment

Local Schema Management

# Clear existing schema and data
make initialize-knowledge-graph-clear

# Apply latest schema changes
make initialize-knowledge-graph

# Verify schema deployment
curl http://localhost:8080/admin/schema

Data Contribution Patterns

The Knowledge Graph supports two data pipeline patterns:

Synchronous Pipeline

Use for: Critical data requiring immediate consistency

Real-time AI Agent state updates
User interaction data
Process execution state

Characteristics:

Strong consistency guarantees
Immediate availability
Higher latency tolerance required

Asynchronous Pipeline

Use for: Bulk data processing and analytics

Historical conversation data
Training data ingestion
Background embeddings generation

Characteristics:

Eventual consistency
Higher throughput
Lower resource impact

Currently only synchronous pipeline is supported

KAG RPC Interface

The Knowledge Graph provides a gRPC interface for cross-language compatibility:

KAG Service Definition

service KnowledgeGraphService {
  // Query operations
  rpc Query(QueryRequest) returns (QueryResponse);
  rpc QueryStream(QueryRequest) returns (stream QueryResponse);
  
  // Mutation operations  
  rpc Mutate(MutateRequest) returns (MutateResponse);
  rpc BatchMutate(BatchMutateRequest) returns (BatchMutateResponse);
  
  // Schema operations
  rpc GetSchema(SchemaRequest) returns (SchemaResponse);
  rpc UpdateSchema(UpdateSchemaRequest) returns (UpdateSchemaResponse);
}

message QueryRequest {
  string query = 1;           // GraphQL query
  map<string, string> variables = 2;
  string tenant_id = 3;
}

Client Integration Patterns

Interface-Based Query Resolvers:

public interface ConversationRepository {
    // Query methods
    Optional<Conversation> findById(String id, String tenantId);
    List<Conversation> findByUserId(String userId, String tenantId);
    Page<Conversation> findRecent(String tenantId, Pageable pageable);
    
    // Mutation methods
    Conversation save(Conversation conversation);
    void delete(String id, String tenantId);
    
    // Graph traversal methods
    List<Message> getConversationMessages(String conversationId);
    List<AIAgent> getParticipatingAgents(String conversationId);
}

@Component
public class DGraphConversationRepository implements ConversationRepository {
    // DGraph-specific implementation
}

Benefits of Interface-Based Approach:

Database Agnostic: Easy to switch between graph databases
Testable: Mock implementations for unit testing
Maintainable: Changes confined to specific implementations
Consistent: Same semantics across different backends

Multi-Database Support Strategy

Direct query translation between graph databases (DGraph ↔ Neo4j ↔ JanusGraph) is not practical due to fundamental differences in query languages and capabilities.

Recommended Approach:

Abstract Business Logic: Use repository interfaces for domain operations
Database-Specific Implementations: Separate implementation for each graph DB
Consistent Data Models: Maintain same semantic meaning across databases
Configuration-Based Selection: Choose database implementation at runtime

Database Selection Config

knowledge-graph:
  provider: dgraph  # dgraph | neo4j | janusgraph
  dgraph:
    endpoint: "http://dgraph-alpha:8080"
  neo4j:
    uri: "bolt://neo4j:7687"
  janusgraph:
    hosts: ["cassandra:9042"]

Next Steps

After successfully deploying the Knowledge Graph service:

Schema Development

Learn how to contribute schemas and follow development guidelines

AI Agent Integration

Configure AI Agents to use the Knowledge Graph for state management

RAG Configuration

Set up Retrieval-Augmented Generation with vector search

Multi-Agent Setup

Enable collaboration between multiple AI Agents

Data Migration

Migrate existing data to the Knowledge Graph

Monitoring Setup

Configure comprehensive monitoring and alerting

Need Help? Check the troubleshooting section above or contact the FlowX.AI support team for assistance with your Knowledge Graph deployment.

Microservices

Plugins

Observability

Microservices access rights

Plugins access rights

​Overview

​Key Capabilities

AI Agent State Management

RAG Support

Multi-Agent Collaboration

Connected Data

​Infrastructure Prerequisites

​Configuration Parameters

​Core Service Configuration

​DGraph Cluster Configuration

​Integration Configuration

​Security Configuration

​Deployment

​Docker Compose Deployment

​Kubernetes Deployment

​Performance Benchmarks

Read Performance

Write Performance

​Test Data Scale

​Health Checks and Monitoring

​Service Health Endpoints

​Key Metrics to Monitor

​Schema Management

​Core Schema Types

​Troubleshooting

​Security Best Practices

​Backup and Recovery

​Automated Backup (DGraph Enterprise)

​Export/Import (DGraph OSS)

​Developer Guidelines

​Schema Contribution Standards

​Schema Deployment Process

​Data Contribution Patterns

Synchronous Pipeline

Asynchronous Pipeline

​KAG RPC Interface

​Client Integration Patterns

​Multi-Database Support Strategy

​Next Steps

Schema Development

AI Agent Integration

RAG Configuration

Multi-Agent Setup

Data Migration

Monitoring Setup

Overview

Key Capabilities

Infrastructure Prerequisites

Configuration Parameters

Core Service Configuration

DGraph Cluster Configuration

Integration Configuration

Security Configuration

Deployment

Docker Compose Deployment

Kubernetes Deployment

Performance Benchmarks

Test Data Scale

Health Checks and Monitoring

Service Health Endpoints

Key Metrics to Monitor

Schema Management

Core Schema Types

Troubleshooting

Security Best Practices

Backup and Recovery

Automated Backup (DGraph Enterprise)

Export/Import (DGraph OSS)

Developer Guidelines

Schema Contribution Standards

Schema Deployment Process

Data Contribution Patterns

KAG RPC Interface

Client Integration Patterns

Multi-Database Support Strategy

Next Steps