--- title: "Circuit breakers, and the case where they make things worse" date: "2022-04-01T05:00:00.000Z" categories: "Cloud native" canonical: "https://xraph.com/writing/circuit-breakers-and-when-they-lie" --- # Abstract A breaker converts a slow failure into a fast one, which is usually what you want. Here is the configuration that turns a partial outage into a total one. ## What a breaker is for A downstream service starts timing out. Without a breaker, every request waits for the timeout, holds a connection and a goroutine while it waits, and the caller's resources fill with requests that were always going to fail. That is how one slow dependency takes down a healthy service [1]. A breaker watches the failure rate, opens when it crosses a threshold, and fails immediately while open. Periodically it allows a probe through, and if that succeeds it closes again. The value is that failure becomes cheap, which keeps the caller alive. ## The configuration that hurts Consider a service with ten replicas of a dependency, one of which is unhealthy. Roughly ten per cent of requests fail. If the breaker's threshold is below that, and the breaker is per dependency rather than per target, it opens against the whole dependency, including the nine healthy replicas. A ten per cent failure has become a one hundred per cent failure, and the system has done it to itself. The fix is that the breaker must sit at the same granularity as the routing decision. If you load balance across targets, break per target, and let the load balancer route around the open one. That is what I ended up doing in both gateways I have worked on, and it is the single most important configuration decision in the component. ## Three more ways to get it wrong **Counting the wrong failures.** A 404 is not a failure of the dependency. A 400 is a failure of the caller. Counting either toward the breaker means a client sending malformed requests can open the breaker for every other client. **Thresholds by count rather than rate.** Five failures means one thing at ten requests per second and something entirely different at a thousand. A rate over a window is what you want, with a minimum sample size so that the first two requests after startup cannot open it. **Synchronised recovery.** If every replica opens its breaker at the same moment and probes on the same interval, they all probe together, and the recovering dependency receives a simultaneous burst from every caller. Jitter on the probe interval is not optional. ## Where a breaker is the wrong tool A breaker responds to a dependency that is failing. It does nothing for a dependency that is succeeding slowly and under-provisioned, where the right response is backpressure and load shedding rather than an open circuit [2]. It also does nothing for a partition, where both sides are healthy and cannot see each other. The breaker will open, correctly, and the underlying decision about what to do while partitioned is the availability and consistency trade-off that no breaker configuration resolves [3]. ## Observability A breaker that opens without emitting anything is a mystery generator. Every state transition should produce a metric with the target and the reason, because the question during an incident is always which breaker opened first, and that is only answerable if the transitions are recorded. # References 1. Michael T. Nygard, "Release It! Design and Deploy Production-Ready Software", Pragmatic Bookshelf, 2nd edition, 2018 2. Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy, "Site Reliability Engineering: How Google Runs Production Systems", O'Reilly Media, 2016 3. Seth Gilbert, Nancy Lynch, "Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services", ACM SIGACT News, vol. 33, no. 2, pp. 51-59, 2002 --- *Unpublished draft. Not peer reviewed and not submitted to a venue.*