Best sampling strategy for opentelemetry & jaeger architecture

[coder-1111]

Hi,

Iam working on setting up a tracing tool. The data flow would be - application -> otel-collector-> jaeger.
I want to enable sampling in my setup. These are the ways i have explored -

1. Sampling from application side - parentbased_traceidratio
2. Sampling on otel-collector level - probabilisticsamplerprocessor (which would work same as 1st point)
3. I'm not sure what can be done from jaeger side

I have explored about tail based sampling. I see that there is a processor for that usecase on otel-contrib repo - https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor. But it doesn't support multiple instances of otel-collector.

Please suggest me a best way to implement sampling for my setup. It would be great if you can suggest something related to tail based sampling
Note- I'm using autoinstrumentation

[bogdandrutu]

/cc @jpkrohling

[jpkrohling]

Sampling from application side - parentbased_traceidratio

Use this if you cannot afford to send data from your application to a collector somewhere.

Sampling on otel-collector level

I prefer this one, as you can radically change your strategy without touching the applications.

I'm not sure what can be done from jaeger side

Jaeger supports sampling at the collector side as well. Look at the flag --downsampling.ratio here: https://www.jaegertracing.io/docs/1.25/cli/#jaeger-collector

I have explored about tail based sampling. ... But it doesn't support multiple instances of otel-collector.

If you are interested in tail-based sampling, you can have a setup like this:

application -> load balancer -> collector (tail-based sampling) -> Jaeger

The load balancer is a collector instance with the load balancing exporter.