A shared library change went out by hand. It passed the build. It passed typecheck. It passed
sixty-nine unit tests. It then crash-looped three applications simultaneously, because the failure
only existed in the bundled output: a configuration object that behaved differently once bundled,
throwing at startup with config.get is not a function.
Worse than the outage was the recovery. All three had been deployed together, and the previous image tag had been overwritten, so there was nothing to roll back to.
What the deploy does now
Every release runs the same path, one application and one part at a time:
- Build on the host that will run it. Not on a laptop with different memory and a different
- architecture.
- Smoke-boot the actual image. Start the image that was just built, in isolation, and wait for
- it to answer. This is the step that would have caught the outage: the bundle only breaks when the
- bundle runs.
- Tag what is currently running as the rollback. Before anything is replaced.
- Deploy, then verify. The container stays up for a probe window with zero restarts; it is
- running the image just built, not a cached one; health returns 200; an unauthenticated request to
- a protected route still returns 401; no error lines in the log.
- Roll back automatically if any of that fails.
The part people skip
The script refuses to deploy an API whose migrations disagree with the live database, and it names the ones that are missing. Applying them stays a deliberate act, after a backup.
This matters because a missing migration is invisible to every check above. The container starts. Health returns 200. Auth works. And then every real query fails, because the column the code expects is not there. Passing a smoke test while being fundamentally broken is exactly the failure mode that deserves a guard.
It also refuses a dirty tree
If the working copy has uncommitted changes, the deploy stops. Whatever ships must be the thing in version control — otherwise the rollback tag points at a commit that never described what was actually running.
The general shape
Every one of these rules came from an incident, not from a best-practices list. That is the useful pattern: when something breaks, add the check that would have refused it, and put the reason in the script's own comments so the next person — usually you, a year later — can see what it is defending against.