Data Streams
Data Streams is an upcoming feature. The functionality described below reflects the planned capabilities.
Data Streams provides end-to-end visibility into asynchronous data pipelines, helping you monitor message queues, streaming platforms, and event-driven architectures.
Overview
Modern applications increasingly rely on asynchronous messaging systems like Kafka, RabbitMQ, and cloud-native event buses. Data Streams helps you:
- Track message flow — Visualize how data moves through your pipeline from producers to consumers
- Measure end-to-end latency — Understand the total time from message production to consumption
- Detect bottlenecks — Identify slow consumers, lagging partitions, and throughput bottlenecks
- Monitor queue health — Track queue depth, consumer lag, and processing rates
Key Concepts
Pipeline
A data pipeline represents the path a message takes from producer to consumer, potentially through multiple intermediate stages (e.g., Producer → Kafka → Consumer → Database).
Latency Breakdown
End-to-end latency is broken down into stages:
- Produce latency — Time for the producer to send the message
- Queue latency — Time the message sits in the queue waiting to be consumed
- Consume latency — Time for the consumer to process the message
Throughput
Message throughput measured at each stage of the pipeline.
Planned Capabilities
Pipeline Map
Visual topology showing data flow through your messaging infrastructure:
- Producers and consumers as nodes
- Queues/topics as intermediate nodes
- Edge labels showing throughput and latency
Stream Metrics
| Metric | Description |
|---|---|
| Messages/sec | Message throughput at each pipeline stage |
| End-to-end latency | Total time from production to consumption |
| Consumer lag | Number of messages waiting to be processed |
| Error rate | Failed message processing rate |
| Queue depth | Current number of messages in the queue |
Supported Platforms
- Apache Kafka — Topics, partitions, consumer groups
- RabbitMQ — Queues, exchanges, bindings
- AWS SQS/SNS — Queue metrics and message tracing
- Google Pub/Sub — Topic and subscription monitoring
- Redis Streams — Stream length and consumer group lag
Alerting
Set alerts on critical pipeline metrics:
- Consumer lag exceeding thresholds
- End-to-end latency spikes
- Message processing failures
- Queue depth anomalies
Getting Started
Data Streams will be accessible from the Data Streams item in the sidebar once the feature is generally available. Stay tuned for updates.