Configuration that reloads without breaking things

#Two kinds of setting

Hot reload is usually implemented as one mechanism: a file changes, watchers fire, values update. That treats every setting as though it has the same relationship to running work, and settings do not.

A log level can change between two log lines and nothing notices. A feature flag can change between two requests. A connection pool size cannot change between acquiring a connection and releasing it without somebody holding a handle to a pool that no longer exists at that size. A listen address cannot change at all without a restart, because the socket is bound.

#The incident

My first implementation fired every watcher immediately. Somebody adjusted a pool size in a running service and produced a class of connection error that took most of a day to understand, because the failure appeared in requests that had started before the change and finished after it.

The logs showed connection failures against a healthy database. Nothing in them mentioned configuration, because from the pool's point of view it had simply been resized while in use.

#The design that replaced it

Reload does not apply a change. It offers one, to the component that owns the setting, which answers with one of three things: applied, needs a drain, or refused until restart.

1type Reloadable interface {
2    // Returning ErrNeedsRestart leaves the running value in place and records
3    // that the desired state and the actual state now differ.
4    ApplyConfig(ctx context.Context, next Config) error
5}
Copy

The third answer is the one that matters. A component that cannot safely change a value at runtime should say so rather than pretending, and the operator should be able to see that the running configuration differs from the desired one. Silently ignoring a change is worse than refusing it, because the operator believes the change took effect.

#Where the boundary belongs

This is a module decision in the sense Parnas describes [3]. Whether a value can change at runtime is a fact about the component that uses it. Putting that knowledge in the configuration library means the library has to know about pools, sockets and consumers, which is exactly the coupling module boundaries exist to prevent.

#The drain case

The middle answer is the interesting one. A component that can adopt a new value, but not for work already in flight, needs to run both configurations briefly. A pool can build a second pool at the new size, direct new acquisitions to it, and retire the old one when its last handle is returned.

That costs memory during the transition and it is the only correct answer for stateful components. The pattern is the same one used for connection draining during deployment [1], and it is worth implementing once in a shared place rather than five times badly.

#What to expose

Three things, and services that expose all three are much easier to operate [2]. The desired configuration, meaning what the source says. The running configuration, meaning what components actually hold. And the difference, which should normally be empty and is the first thing to look at when behaviour does not match the file.