Observability is the capacity to infer the internal state of a system by analyzing its external outputs. In software development, this entails understanding the internal workings of a system through its telemetry data, which comprises traces, metrics, and logs.
OpenTelemetry is an observability framework and toolkit for generating and managing telemetry data, including traces, metrics, and logs. It is vendor-agnostic and compatible with various observability backends like Jaeger and Prometheus. Unlike observability backends, OpenTelemetry focuses on the creation, collection, and export of telemetry data, leaving storage and visualization to other tools.
Tracing with Open Telemetry is availabile starting with FlowX.AI v.4.1.0 release.
Our monitoring and performance analysis system leverages OpenTelemetry for comprehensive tracing and logging across our microservices architecture. By integrating with Grafana and other observability tools, we achieve detailed visibility into the lifecycle of requests, the performance of individual operations, and the interactions between different components of the system.
OTEL Collectors are designed in a vendor-agnostic way to receive, process and export telemetry data. More information about OTEL Collectors, you can find here.
Custom Spans: These were created for methods important to the business flow and enriched with business attributes such as fx.type, fx.methodName, fx.processInstanceUuid, and others.
Custom BUSINESS Spans: Create spans for business events.
Spans now include custom FlowX attributes (e.g., node names, action names, process names, instance UUIDs), which ccan be used for filtering and searching in traces.Here is the full list of included custom FlowX span attributes:
We recommend to use Grafana, but any observability platform compatible with open telemetry standards can be used.
Grafana integrates with tracing backends such as Tempo (for tracing) and Loki (for logging), allowing us to visualize the entire lifecycle of a request. This includes detailed views of spans, which are the basic units of work in a trace. By using Grafana, we can:
View Trace Trees: Grafana provides an intuitive UI for viewing the hierarchy and relationships between spans, making it easier to understand the flow of a request through the system.
Filter and Search: Use Grafana to filter and search spans based on custom attributes like fx.processInstanceUuid, fx.nodeName, fx.actionName, and others. This helps in pinpointing specific operations or issues within a trace.
Error Analysis: Identify spans with errors and visualize the stack trace or error message, aiding in quick troubleshooting.