> For the complete documentation index, see [llms.txt](https://stephen-tsoi.gitbook.io/stephen-tsoi-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://stephen-tsoi.gitbook.io/stephen-tsoi-docs/opentelemetry/from-missing-events-to-complete-visibility-tracing-enterprise-transactions-with-opentelemetry/day-17-34-sampling-tail-sampling.md).

# Day 17/34 Sampling – Tail Sampling

**Tail Sampling** offers a unique perspective on observing real behavior in logging systems. It involves capturing 100% of error or timeout traces and a percentage (N%) of success traces, while the Matrix encompasses all trace information.

In the backend, designed with two tiers of collectors, the initial layer constructs the Matrix using all trace data and directs trace spans to the second-tier collector based on specific configurations. The second tier temporarily stores all spans sharing the same trace id before forwarding them to the APM backend according to predefined rules:

1. Success cases are sent based on N% sampling.
2. Failures are logged at 100%.
3. Timeouts are also captured entirely at 100%.

On the client side, all trace spans are consistently transmitted to the APM backend without sampling.

However, challenges arise:

* Real-time trace visibility is limited due to templated storage on the second-tier collector.
* The second-tier collector demands robust servers with substantial memory capacity to retain spans within the same trace id for a defined period.
* Scaling the second-tier collector is complex, requiring redeployment of configurations across all first-tier collectors.

During a visit to an APM vendor's development center in the first week of October 2025, it was revealed that they have implemented a two-tier collector infrastructure on their APM backend to optimize disk usage. Additionally, they recommended utilizing the APM agent as the first-tier collector and distributing trace spans to the second tier based on the trace id using a modulo calculation (\[trace id] % \[number of collectors]).

<figure><img src="https://2617374589-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FcQN1DZY6gJZPxlsQf9Re%2Fuploads%2FvAfgKFioL36sRBguQtwY%2Fimage.png?alt=media&amp;token=83cae4b1-0ea6-4880-9150-ebc9575d7142" alt=""><figcaption></figcaption></figure>

## 🚀 Let's Connect Beyond GitBook!

If you found this article helpful, you can find more of my technical insights, daily discussions, and deep dives across these platforms:

* **Read more of my work:** Check out my articles on [dev.to](https://dev.to/stephen_tsoi_5b2c4055f3a9) and [Hashnode](https://stephentsoi.hashnode.dev/).
* **Join the daily conversation:** Connect with me directly on [LinkedIn](https://www.linkedin.com/in/stephen-tsoi-16309730/).

***

#### 📬 Stay Ahead of the Curve

Enjoyed this piece? I break down complex technical topics into bite-sized, actionable insights every week.

👉 **Subscribe to my** [**LinkedIn Newsletter**](https://www.linkedin.com/build-relation/newsletter-follow?entityUrn=7487299517642612736) to never miss an update and get the latest articles delivered straight to your feed!


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://stephen-tsoi.gitbook.io/stephen-tsoi-docs/opentelemetry/from-missing-events-to-complete-visibility-tracing-enterprise-transactions-with-opentelemetry/day-17-34-sampling-tail-sampling.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.