Health checks that mean something

#The endpoint that does nothing

Plenty of services expose a health endpoint that returns 200 unconditionally. If the process is running it answers, and if the process is not running it does not, which the orchestrator could already determine by looking at the process.

It feels like diligence and it carries no information.

#Two questions, not one

Kubernetes separates liveness from readiness because the two failures need opposite responses [2].

Liveness asks whether the process should be killed and restarted. The correct answer is no for almost everything. A database that is temporarily unreachable is not a reason to restart your process, because restarting will not reconnect it any faster and a restart loop across every replica during a database incident turns a degradation into an outage.

Readiness asks whether this instance should receive traffic right now. That is where dependency state belongs. A process with an empty connection pool is alive and not ready, and the correct response is to route around it until it is.

Conflating them is how a brief dependency blip becomes a cascading restart. Nygard's discussion of failure propagation is the clearest treatment of that dynamic [3].

#Discovery rather than registration

The mechanical part I care about is who maintains the list of checks. If health is a list somewhere, the list falls out of date, because adding a component and remembering to register its check are two actions and only the first is required to make the feature work.

Making it discovery instead solves it. Any component that implements the health interface is found by the container and appears in the report. There is no list, so there is nothing to forget.

#What a useful report contains

Per component: a name, a state, how long the check took, and a message when the state is not healthy. The aggregate is derived rather than stored, and degraded is a real state rather than a rounding error toward healthy or unhealthy.

The duration matters more than people expect. A check that has started taking 900 milliseconds is telling you something before it starts failing, and it is the earliest signal you will get. Service level objectives are built on exactly this idea, that a slow success is a partial failure [1].

#The trap

Do not put an expensive query in a health check. It runs on a timer, on every replica, forever. I have seen a readiness probe that ran a count over a large table, and it was a meaningful fraction of the database load in a system that was otherwise healthy.

Check that the connection works. Do not check that the data is correct.