---
title: "Zero downtime is a property of the version you are replacing"
date: "2022-09-01T05:00:00.000Z"
categories: "Cloud native"
canonical: "https://xraph.com/writing/zero-downtime-is-a-property-of-the-old-version"
---

# Abstract

Deploy strategies get the attention, and most failed deploys I have seen failed because the outgoing version could not shut down cleanly or the schema change was not backward compatible.

## The attention goes to the wrong half

Rolling, blue-green and canary describe how the new version arrives. All three assume the old version leaves cleanly and that both versions can coexist. When a deploy drops requests, it is usually one of those two assumptions that was false.

## Leaving cleanly

A process receiving SIGTERM has work in flight. Exiting immediately drops it. The correct sequence is specific and most frameworks do not do all of it.

1. Fail the readiness probe, so the load balancer stops sending new requests. This is first, and the gap between failing readiness and the balancer noticing is real, which is why a short sleep here is a legitimate technique rather than a hack.
2. Stop accepting new connections while continuing to serve open ones.
3. Wait for in-flight requests, with a deadline shorter than the orchestrator's grace period.
4. Stop background workers, letting the current unit of work finish rather than abandoning it.
5. Close resources in reverse dependency order, so nothing is released while something else still needs it.

Step five is why I care about lifecycle graphs. Shutdown ordering derived from a declared graph is correct by construction, and shutdown ordering maintained by hand drifts every time somebody adds a component.

## Coexisting

During any rolling deploy, both versions are live and serving. Everything they share has to work for both.

Schema changes are where this bites. Adding a nullable column is safe. Renaming one is not, because the old version selects a column that no longer exists. The rename becomes three deploys: add the new column and write both, migrate readers, then drop the old column once no running version references it. That is tedious and it is the only version that works [1].

The same applies to message formats. A consumer of the old version must tolerate a message produced by the new one, which usually means additive-only changes for a full deploy cycle.

## What the evidence says about frequency

The intuition is that deploying less often is safer. The data from the DevOps research programme points the other way: teams that deploy more frequently have lower change failure rates and recover faster [2]. Small changes are easier to reason about and easier to reverse, and a rollback of one change is a decision rather than an investigation.

## The rollback question

A rollback plan that has never been executed is a hypothesis. The specific thing that invalidates it is a migration that ran during the deploy: the code can go back and the schema cannot, and now the old version is running against a shape it does not understand.

Keeping migrations backward compatible for one version is what makes rollback real [3] [4]. It is more work and it is the difference between a rollback and an incident.

# References

1. Jez Humble, David Farley, "Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation", Addison-Wesley, 2010

2. Nicole Forsgren, Jez Humble, Gene Kim, "Accelerate: The Science of Lean Software and DevOps", IT Revolution Press, 2018

3. Michael T. Nygard, "Release It! Design and Deploy Production-Ready Software", Pragmatic Bookshelf, 2nd edition, 2018

4. Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy, "Site Reliability Engineering: How Google Runs Production Systems", O'Reilly Media, 2016

---

*Unpublished draft. Not peer reviewed and not submitted to a venue.*