Skip to content
sumanth.aiAN ENGINEER’S WORKSHOP
← All articles

Security & Platform / 2 MIN READ

A rollback button is only the beginning

Release recovery needs compatible data, a known-good artifact, and a decision rule.

Rolling back an application image is straightforward compared with rolling back everything the application touched. A release can change data, queue messages, cache entries, credentials, and assumptions made by another service.

I would review recovery as a separate design problem. “Redeploy the previous version” is a useful step, but it is not a complete plan.

Name the state that survives the deployment

For an illustrative release that adds a database field, ask whether both the old and new application can tolerate the transition. An additive migration can make that easier. A destructive rename or removal can make the previous application unusable even when its image is available.

List the other persistent effects as well: events already published, jobs already scheduled, and cache values written in a new format. Some need a compensating action or a forward fix rather than a reverse deployment.

Keep the observation window honest

A canary with no meaningful traffic tells you little. Decide which journeys must be exercised and whether the sample is large enough to reveal the failure you care about. Compare the candidate with a useful baseline, not just a dashboard that happens to be green.

User-facing error rate and latency can inform the decision, but they need a denominator and a time window. Google’s SRE guidance on SLO-based alerting is useful here: error-budget consumption can connect the response to the impact rather than a raw exception count.

Write the decision before the incident

My release note would include the artifact identifier, previous known-good version, migration compatibility, rollback steps, verification steps, and who can stop the rollout. Include the conditions under which rollback would make things worse.

For example: if the application is failing but no incompatible data has been written, reverting the artifact may be reasonable. If the old version cannot read the new state, stopping writes and applying a forward fix may be the safer path. That choice belongs in preparation, not improvisation.

Finally, exercise the recovery path in a representative environment. A runbook that has never been followed is a hypothesis. The point of rehearsal is to discover which permissions, artifacts, and assumptions are missing while there is still time to fix them.

Try it / Plan a staged rollout

A small example to explore alongside this article. Inputs stay in this page.

Related tools: Plan a staged rollout ↗

Was this useful?

Account-free. Your response stays in this browser; it is not sent to a server.

REFERENCES & FURTHER READING

Official documentation and primary sources behind the technical details. Examples in this article are illustrative.

KEEP READING

Security & Platform / 2 min

Put the secure path where developers already work

Read article ↗
Browse the whole notebook →