> For the complete documentation index, see [llms.txt](https://stephen-tsoi.gitbook.io/stephen-tsoi-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://stephen-tsoi.gitbook.io/stephen-tsoi-docs/rethinking-opentelemetry-tail-sampling/8.-advanced-tail-based-sampling-solution-by-eda-design.md). # 8. Advanced Tail-Based Sampling Solution (By EDA design) *The EDA-based architecture introduced in this book is a proposed enterprise implementation pattern and is not part of the official OpenTelemetry Collector reference architecture.* The traditional two-tier OpenTelemetry (OTel) collector architecture can effectively address end-to-end tracing sampling challenges. However, it also introduces significant complexity and is difficult to implement in practice. One of the primary hurdles is the requirement for extremely powerful infrastructure—often necessitating high-performance servers with very large memory capacity—to handle traffic at tens of thousands of spans per second. A key limitation of this traditional design lies in its point-to-point integration between the tier-1 and tier-2 collectors. This tight coupling forces the tier-2 collectors to independently manage resilience, scalability, and fault tolerance, which adds operational burden and limits flexibility. In this section, we propose a modern alternative: replacing point-to-point communication with a messaging-based Pub/Sub integration. By decoupling the two tiers through a messaging layer, this approach improves scalability, enhances resilience, and redistributes complexity from collectors to the messaging infrastructure to solve the major challenge of the traditional two-tier OpenTelemetry collector design. The concept is similar to the Kafka-based buffering approach commonly used in large-scale tracing platforms, where trace collection is decoupled from downstream processing through an intermediary messaging layer. The proposed architecture introduces an alternative approach to tail-based sampling by externalizing trace buffering and coordination functions from the collector layer to the messaging infrastructure. This design is intended to reduce collector state management requirements while providing additional flexibility for scalability, resiliency, and operational deployment. The following sections describe the architecture, implementation options, benefits, and associated trade-offs. ### 8.1 Architecture Design The architecture is built around an event mesh that provides both reliable trace transport and temporary persistence. The event mesh is a virtual messaging infrastructure built on top of the underlying network. It is formed by a collection of interconnected message buses that work together as a unified event distribution layer. The message bus is responsible not only for routing data between tier-1 and tier-2 collectors but also for providing a persistent storage layer for the incoming trace. This design removes the need for tier-2 collectors to cache all spans belonging to the same trace. Instead, the message bus temporarily retains the data, allowing tier-2 collectors to consume and process it in batches before forwarding it to the APM backend.

Figure 8-1: Overall Architecture

Decoupling – this solution uses fan-out mechanisms to replace point-to-point integration. Collector decoupling is not only on the geolocation, but also on the system interface, the size of payload and event flow. Resilient – both tier-1 and tier-2 collectors adopt a stateless, active-active design, ensuring high availability and fault tolerance. The message bus is deployed in an active-standby configuration to support high availability and disaster recovery requirements. Trace data is persistently stored in the message bus until it is successfully consumed by the tier-2 collectors. These collectors process and send trace data to the APM backend in batches and only acknowledge the messages after successful delivery, ensuring reliable processing and preventing data loss. Scalability – the tier-1 collectors are stateless and can scale horizontally based on the throughput of upstream applications and third-party product workloads. Similarly, the tier-2 collectors support horizontal scaling, driven by the depth of the message queue within the message bus. This allows the system to dynamically adjust to workload fluctuations and maintain processing efficiency. Performance – Enterprise-grade message bus platforms are designed to support high-throughput data exchange. Depending on factors such as product capabilities, deployment topology, persistence settings, hardware resources, and workload characteristics, these platforms may achieve throughput ranging from thousands to hundreds of thousands of messages per second. When combined with batch processing by tier-2 collectors, the messaging layer can help reduce processing overhead and improve overall system scalability and efficiency. There are three major components in this architecture design * The tier-1 OpenTelemetry collectors are responsible for collecting traces, metrics, and logs from source systems, including applications and third-party platforms such as API gateways and message buses. These collectors then publish the collected data to the message bus using different topic names to categorize the data streams. * The message bus acts as the central data distribution layer, routing incoming data to single non-exclusive queues. These queues provide persistent storage for the trace data, ensuring that it is retained until it is consumed and acknowledged in batches by the tier-2 OpenTelemetry collectors. This persistence mechanism enhances reliability and decouples data ingestion from downstream processing. * The tier-2 OpenTelemetry collectors follow a standalone, stateless design and subscribe to the relevant data streams from the message bus. They process the incoming telemetry and apply sampling policies, where successful traces are sampled before being sent to the APM backend, while failed traces are forwarded at a 100% rate to ensure complete visibility into errors and issues. Compared with traditional tail-based architecture, this solution only requires enhanced communication between the tier-1 and tier-2 collectors to support the message bus. ### 8.2 Infrastructure Design Similar to a traditional two-tier OpenTelemetry (OTel) Collector architecture, the tier-1 collectors are distributed across application hosts or deployed on dedicated collector servers in different network segments isolated by firewalls. These collectors connect to a message bus configured in an active-standby topology to provide high availability (HA) and disaster recovery (DR) capabilities. The tier-2 collectors are deployed within the same network segment and consume data from the message bus. As a result, there is no need to deploy a dedicated load balancer in front of the tier-2 collectors, since message distribution and failover are handled by the message bus infrastructure.

Figure 8-2: Overall Infrastructure

### 8.3 Solution Design This solution adopts a two-tier collector design, with communication between the collector tiers facilitated through the message bus. Redis is optionally used as an in-memory cache to store global trace IDs and their associated metadata. This solution also introduces some extra optional services to decouple the function from the tier-2 collector. The solution consists of three core components and three optional components, as described below. **Message Bus** Within the event mesh, the message bus serves as the fundamental building block. Its role extends beyond simply transporting trace data between tier-1 and tier-2 collectors. By leveraging messaging capabilities such as routing, filtering, buffering, and protocol abstraction, the message bus processes and distributes messages to the appropriate consumers. This allows the message bus to distribute trace data efficiently while reducing the complexity of the application logic required by the tier-2 collectors, thereby improving the scalability and maintainability of the overall solution. **Collector** The OpenTelemetry (OTel) Collector consists of three main components: receivers, processors, and exporters. In this architecture, the tier-1 collector receives traces from application SDKs or third-party products through its receivers. The traces are then pre-processed by the processors before being published to a message bus and forwarded to the tier-2 collector. The tier-2 collectors subscribe to trace data from the message bus, perform additional processing as required, and batch all spans belonging to the same trace before exporting them to the APM backend. The OTLP receiver and exporter support gRPC and HTTP, and the contrib repository provides messaging-specific components — including a Kafka receiver and exporter, and a Solace receiver for broker-generated trace events. Depending on the chosen message bus, integration may therefore require configuration of existing components rather than custom implementations; a custom exporter is required only where no component exists for the target broker. In addition, the processor component must be enhanced to read and update trace metadata and global configuration parameters, such as sampling-strategy settings, from Redis or another persistent data store. In many deployments, organizations use vendor-specific agents instead of tier-1 OpenTelemetry Collectors. As a result, it is necessary either to enhance the vendor agents to publish trace data to the message bus directly or to introduce an additional collector layer that acts as the tier-1 collector and receives telemetry data from these agents before forwarding it to the message bus. **Register Service (Optional)** **Redis (Optional)** #### 8.3.1 Dead Letter Queue Design This design relies on the Dead Letter Queue (DLQ) functionality of the message bus. The tier-1 collector sends trace spans to the message bus, and the message can then be retrieved from the DLQ after 30 seconds.

Figure 8-3: Dead Letter Queue Design

**Tier-1 Collector Processing** The tier-1 collector ingests trace spans from applications and third-party products and publishes them to dedicated message-bus topics (otel\_trace/\[trace\_id]). Each message (trace span) must include the following mandatory header fields: * Span Type (Root/Child) * Trace ID **Message Bus Design** A shared non-exclusive data queue subscribes to all trace topics and enforces a 30-second TTL on queue level. Any unconsumed message is automatically forwarded to Dead Letter Queue upon expiration. The data queue’s subscription rule is as follows:

Table 8-1 DLQ Queue Solution Queue Subscription Rules

QueueOriginal Subscription
Data Queueotel_trace/>
**Tier-2 Collector Processing** The tier-2 collector monitors Dead Letter Queue and uses the first expired root span as a trigger to process the corresponding trace. It retrieves all spans sharing the same trace ID from both the active queue and the dead message queue through selector-based filtering. The complete trace is then evaluated against the configured sampling rules. Qualified traces are batched and forwarded to the APM backend, while unqualified traces are discarded. **Queue TTL + Dead Message Queue Equivalent to Buffer Window** This design leverages the queue Time-to-Live (TTL) and Dead Letter Queue (DLQ) to achieve the same objective as the buffer window used in traditional tail-based sampling. When a tier-2 collector consumes a message from the control queue, it can be assured that all spans generated within the preceding 30-second window have already been persisted in the data queue. The buffer duration is configurable and can be adjusted dynamically, with changes taking effect immediately. #### **8.3.2 Delay Queue Design** #### 8.3.3 Message Bus Delay Queue + Redis #### 8.3.4 Recommended EDA Solution #### 8.3.5 Advantages of an EDA Design #### 8.3.6 Risk #### 8.3.7 EDA Option Comparison ### 8.4 Remote Global Parameter Update ### 8.5 Implementation Considerations ### 8.6 Benefit ### 8.7 Trade-Offs ### 8.8 Traditional Two-Tier Collector vs. EDA Solution