Instrument before the incident, not during it

#The pattern

Something is slow. Nobody can say which of six services is responsible. Somebody adds timing logs to the two most likely candidates, deploys, waits for the problem to happen again, and discovers it was a seventh service nobody had suspected.

That cycle costs a day and it repeats, because the instrumentation added during an incident is added to the services that were suspected rather than to the ones that were not.

#What tracing is actually for

A distributed trace answers a question that logs cannot: for this one request, where did the time go, across every process it touched. Dapper described the design constraints clearly, and the important ones are that instrumentation must be ubiquitous and continuously deployed, because a trace with a gap in it is a trace that cannot answer the question [1].

Ubiquitous is the operative word. Nine services instrumented out of ten gives you a trace that ends at the boundary of the tenth, which is very often exactly where the problem is.

#Why it belongs in the framework

If instrumentation is a library that each service chooses to adopt, coverage will be partial, and it will be partial in a way that correlates with which teams were busy. If it is a property of the framework, coverage is total by construction.

Concretely, that means context propagation through the request path without the handler doing anything, spans around outbound calls created by the client the framework provides, and a trace identifier in every log line because the logger reads it from context.

That last one is the cheapest win in the list. Being able to take a trace identifier from an error and find every log line from every service for that request removes most of the guessing.

#Sampling

Tracing every request at high volume is expensive in a way that shows up on a bill. Sampling is necessary, and the naive version, a fixed percentage of requests, samples away the interesting ones, because errors are rare and rare things are what sampling removes.

The approach that works is to decide late: buffer the spans, and keep the trace if it contained an error or exceeded a latency threshold. That costs memory and it keeps the traces you actually want. The OpenTelemetry specification describes the sampling points and their trade-offs [3].

#Metrics and logs still matter

Traces answer questions about one request. Metrics answer questions about aggregate behaviour, and they are what alerts should fire on, because an alert on a single request is noise [2]. Logs carry the detail that neither of the others has room for.

The mistake is treating them as alternatives. They answer different questions and a service that has only one of them will be diagnosed slowly.