Introduction to Devops

Deployment Automation, Rollback, and Zero Downtime Strategies

Published: 2026-08-14

Level: postgraduate

Audience: Postgraduate students of software engineering and delivery

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

Value stream maps — covered in Lecture 3 (The Need for DevOps)
The CI/CD pipeline flow — covered in Lecture 11 (Continuous Integration Best Practices and CI/CD Pipelines)
Continuous delivery and continuous deployment — covered in Lecture 13 (Deployment Pipelines and Continuous Delivery)
Rollback and roll forward — covered in Lecture 13 (Deployment Pipelines and Continuous Delivery)
Canary testing — covered in Lecture 13 (Deployment Pipelines and Continuous Delivery)
Change management and zero downtime — covered in Lecture 2 (ITIL and the Operational Side of DevOps)

Deployment Automation, Rollback, and Zero Downtime Strategies

15.1 Human-Free Deployment

15.1.1 What Human-Free Deployment Means

Hook. Imagine a software release where no developer, tester, or operations engineer needs to click a button, raise a ticket, or pass a message — the code moves from "checked in" to "live" by itself. Is it possible to make a release that safe, and what does an organization gain from it?

Human-free deployment is the result of automating the deployment and release process, which was the subject of the previous session. Once the release pipeline is automated, the biggest thing an organization looks for is a drop in the issues reported from manual mistakes. The release process runs automatically, human intervention is removed, and the chance of manual errors falls sharply. If anything still goes wrong during a release, you can track it down with audit logs: every tool in the chain keeps its own history. For example, the version control system (such as GitHub) and your monitoring tool each maintain audit logs. From those logs you can check where the problem was triggered and work on restoring the service.

Intuition + analogy — removing the handshakes. The professor's picture of the old way is a chain of handshakes: developer hands code to a release team, the release team hands a build to a tester, the tester emails operations to ask for an environment, and every handoff is an invitation to miscommunication and delay. Human-free deployment removes the handshakes: the process is no longer built on handoffs between people; it is built on automations.

Think of the difference between a bank counter and a self-service checkout. At the counter you stand in line, fill forms, and wait for an employee to act on each request; any missing signature stops the whole process. At the self-service checkout the machine runs the same steps for every customer, instantly and without a queue. The analogy breaks in one place: a checkout machine cannot design itself. Someone has to build and maintain the automation — the machine is human-free to operate, never human-free to create.

A second benefit is that the delivery becomes available to everyone. Each and every team member knows what the deployment and release process is, what the release script is, and which environment the code is residing in when the application is released. Developers, testers, and the operations team no longer depend on the ticketing system, which removes a whole layer of human dependency. The team stops facing email threads that ask to get builds deployed so that feedback on production readiness can be gathered.

Formalize. In one sentence: human-free deployment (automated deployment driven by the release pipeline) is the state in which a version of the application is promoted from commit to production by scripts, with people acting only on exceptions. It rests on three promised benefits:

Fewer manual errors — automation replaces error-prone hand-performed steps, and any residual problem is traceable through audit logs (who ran what, on which machine, when, and what the output was).
Self-service delivery — every team member can select a version and an environment and deploy on demand, ending the dependency on ticketing systems and email threads.
A rehearsed release path — because the same automated script runs constantly, the release process is exercised, tested, and perfected long before a real production moment (detailed in section 15.1.2).

A small tester example makes this concrete. With human-free deployment, any tester does not have to depend on the developers or the operations team to ask for an environment or for a particular working version of the application. The tester decides by themselves which version they would like to test and on which test environment they will be testing, and they access whatever they want at that particular time. The sales team gets the same freedom: salespeople can access the latest version of the application to show the killer features to a client in order to crack the deal. They do not have to drop a mail or check whether a developer is available for a demo; they just enter the password for that environment, click on the features, and start using the application. That is the demo.

Worked walkthrough — the tester's morning, end to end. Suppose the team uses Jenkins with a pipeline script and GitHub for version control.

A developer merges code into the main branch at 9:00 AM. The check-in triggers the pipeline: the build runs automatically, unit tests execute, and code analysis (for example SonarQube) produces a report — nobody opens a ticket.
At 9:12 AM the tester opens the deployment tool, sees the list of builds that have passed the automated stages, picks build number 147, chooses the staging environment, and presses the deploy button. The tool deploys that exact build — not "whatever is currently on the server."
At 9:20 AM the salesperson, preparing for a client demo at 11:00 AM, logs into the demo environment (previously just another environment in the same pipeline), opens the freshly deployed version, and clicks through the new feature in front of the client.

Sense-check: in all three steps, no email, no ticketing, no waiting for "the person who knows how to deploy" — the freedom the professor described is real because the pipeline itself does the work.

Assumptions & scope. Human-free deployment only delivers those benefits when:

The process is scriptable and repeatable. If the release still contains steps that can only be done by hand (for example, a manual database edit), those steps remain error-prone and untraceable.
Environments are consistent. The same script must be able to deploy to staging, acceptance, and production; if those environments have drifted apart, the automation will behave differently in each one.
Audit logs are actually kept. The benefit of traceability depends on every tool in the chain recording its history; a pipeline without logging is blind when something fails.

Visual intuition — the release pipeline. Picture a horizontal flow of boxes: Check-in → Build → Unit tests → Code analysis → Acceptance tests → Deploy to staging → Deploy to production. The horizontal axis is time (minutes, not weeks); each box is a stage run automatically by the same script. The landmark to notice is the commit-to-production distance: in a human-free pipeline it shrinks until releases can happen many times a day. One-sentence takeaway: the flow line is straight and unbroken — no box is a human waiting to be asked.

15.1.2 The Release Process Gets Rehearsed, Tested, and Perfected

A third reason the risk of releases drops is that the release process itself gets rehearsed, tested, and perfected. The release process is nothing but a script in which everything is automated: the build process is automated, the testing process is automated, and code analysis runs automatically. The deployment itself is completed with a script such as a Jenkinsfile when you use Jenkins for integration. Because the same script deploys your system to every environment — staging, the capacity and acceptance testing environment, and production — the deployment process is exercised constantly. It gets tested many times in a day: every time a team member checks in code, the deployment script runs again. A process that runs hundreds of times refines itself far faster than a manual process that runs once in a while.

Pitfalls — why "automated" can still go wrong.

Assuming automation means zero errors. Human-free deployment removes manual mistakes; it does not remove design mistakes (a wrong calculation in the script itself) or environment differences. That is exactly why audit logs matter — when a release fails, the logs, not memory, tell you where it was triggered.
One-off "emergency" bypasses. The temptation to patch production by hand to save time quietly reintroduces the untested, unlogged path the pipeline was built to eliminate (dealt with properly in section 15.8).
"It works in our environment" drift. If the acceptance environment differs from production, the script rehearses against a stand-in; teams then discover production-only failures at the worst possible time. The script must run against every environment, not only the convenient ones.
Believing rehearsal happens without check-ins. The rehearsal benefit depends on frequency: the pipeline only refines itself if code is checked in and the script actually runs. A pipeline that nobody commits to is a manual process wearing a costume.

15.1.3 Student Questions and Answers

Q: In the old days we used Visual Source Safe (VSS) as the version control system, and there were no automated deployments at all — it was a complete manual process. We had a lot of checks process-wise, but deploying was still not easy: we had to go through the whole lifecycle, with BMC Remedy tickets and a completely separate implementation team. Production was a complex system, and we as developers did not have access to all the systems.

A: That is exactly the pain that human-free deployment removes. When a specific team alone works on production deployments, you still have to perform the validations when something goes wrong and when you have to rebase the code. With a manual handoff lifecycle like that, deployment becomes challenging and difficult. The checks existed, but the deployment path itself was slow, constrained, and full of handshakes between people. Notice what the student's story proves: process checks alone are not enough. What made the difference was the deployment path itself — who can trigger a release, how long each handoff takes, and how much of the process is automated.

Recap + bridge. Human-free deployment replaces handoffs between people with automations, cutting manual errors, making delivery self-service for every team member, and rehearsing the release path until it is routine. In the next section we walk through the steps an organization follows to actually build such a pipeline.

Real-world & domain connection. The pattern is not theoretical. At Etsy, deployments became so safe and routine that new engineers performed a production deployment on their first day at work, and engineers deployed to production tens of times per day through an internal tool. At Facebook, release engineering pushed the entire server fleet to new code in about twenty minutes with no visible impact to users — deploying so routinely that the company went from weekly to thrice-daily pushes. Both companies depend on the same idea from this section: the release path is automated, logged, and rehearsed constantly, so that releasing stops being a drama and becomes a normal part of daily work. In the DevOps field this is the difference between a release project and a release process: one is an event with people and risk, the other is a script with audit trails.

15.2 Steps to Implement a Deployment Pipeline

15.2.1 The Seven Steps

Hook. You cannot automate a process you have not written down. Before a single script is written, this section asks the surprisingly hard question: what exactly happens to code from the moment it is committed until the moment it serves users — and who is responsible at each step?

The first step is to model your value stream map: create a walking skeleton of your application. That means deciding what language the application is in and which tools and technology you will use to build that application. Everything downstream is built on top of this skeleton.

Purpose — why start with a walking skeleton. A walking skeleton (the thinnest possible end-to-end slice of the application that can be built and deployed, e.g., an application that prints "Hello, world" through the full stack) is the visible spine of your value stream map — the diagram showing every step your work flows through, from idea to the customer. Before automating anything, you map the flow; once the map exists, you automate it in the order below. Everything downstream is built on top of this skeleton, so choices about language, tools, and technology made here constrain every later decision.

From there, the automation runs in this order:

Steps — the automation order, with the rationale for each.

Automate the build process. Compile the source, run it through packaging, and produce a deployable artifact (a jar, a container image, an installable package). Rationale: a build that runs the same way every time removes "it works on my machine" differences between developers' laptops.
Automate the deployment process. Take the artifact produced by the build and place it into an environment by script (for example a Jenkinsfile when you use Jenkins for integration). Rationale: deployment becomes a repeatable, reviewable action instead of a remembered sequence of manual steps.
Automate unit testing. Run the unit tests automatically on every build, so a failing test blocks the flow immediately. Rationale: defects are caught minutes after they are introduced, when the fix is cheapest.
Add code analysis to complete a code-level review and jot down any code-level defects that the analysis can track. Tools like SonarQube scan for coding violations, security weaknesses, and maintainability problems without a human reviewer reading every line. Rationale: automated analysis catches the defects that manual review misses or takes too long to find.
Automate the acceptance tests. Verify that the application actually does what the business asked for — not just that the code runs. Rationale: acceptance tests confirm behavior end to end, so a build that passes unit tests but fails business expectations never proceeds.
Automate the release process: deploy the application to the delivery environment, deliver it to the staging environment, and finally deploy it to production. Rationale: the final, highest-risk step becomes the same scripted action that has already succeeded in every earlier environment.

These are the basic steps of implementing any deployment pipeline.

Trace — a shopping-list app walks the pipeline. Track build #42 of a small to-do application from commit to production.

#	Stage	What runs	Result (real numbers)
0	Walking skeleton	Developer commits the first deployable slice (app + build config + one test)	Repository contains a working skeleton, language and toolchain decided
1	Build	Compile + package	12,480 lines compiled, `app-42.jar` produced in 1 minute 10 seconds
2	Deploy to test	Script places the jar on the test server	Test environment running build 42
3	Unit tests	318 unit tests run automatically	318 passed, 0 failed
4	Code analysis	SonarQube scan	2 code-smell warnings, 0 blockers — noted in the report
5	Acceptance tests	End-to-end tests: add item, mark done, delete item	All 14 scenarios passed
6	Release	Same script promotes build 42 to delivery → staging → production	Build 42 live; the exact artifact that passed every test is the one serving users

Sense-check: each step ran the same artifact and the same script that the previous step had already exercised, so the release to production was the most rehearsed action in the whole chain, not the riskiest.

15.2.2 Why the Same Process Everywhere Lowers Risk

The power of the pipeline is that the exact same process moves the code through every environment. The staging deployment and the production deployment run the identical script, so what you tested is exactly what goes live. Because the script is exercised so frequently — whenever anyone checks in code — defects in the process itself surface early and get fixed quickly. That frequent repetition is the rehearsal benefit from the previous section, and it is the reason an automated pipeline is safer than a manual release, not just faster.

Pitfalls — where the seven steps usually go wrong.

Building, but not deploying, automatically. Stopping after step 1 or 2 leaves the riskiest part (deployment) manual; the pipeline becomes a fancy build machine with a hand-operated release.
Skipping code analysis. Teams often treat analysis as optional; without it, style and security problems accumulate silently until they become production incidents.
Letting environments drift. If staging and production differ (different configuration, different versions of middleware), "what you tested is exactly what goes live" silently becomes false — the identical script runs against different targets.
Never writing the value stream map. Starting automation without the map automates the current handoffs, cementing waste into the pipeline instead of removing it.

Recap + bridge. A deployment pipeline is built by mapping the value stream, creating a walking skeleton, and then automating build, deployment, unit tests, code analysis, acceptance tests, and finally the release — all with one script. That single script is what makes the pipeline safer: the production deployment is the same process already exercised hundreds of times. The next section faces the other side of safety: when a release still goes wrong, how do you get back to the previous version?

Real-world & domain connection. This is the blueprint used by the continuous delivery movement. The textbooks describe the same sequence: document the release process (value stream mapping), then simplify and automate the manual steps — packaging, copying artifacts onto servers, restarting services, running smoke tests, and scripting database migrations. The payoff shows in numbers: the CSG International team doubled their release frequency while daily-deploying to pre-production environments, and saw production incidents drop 91% and mean time to recover drop 80%. In the DevOps field, "walking skeleton first" is also how teams prove a pipeline exists before a single feature is built — prioritizing pipeline creation over business value in the very first iteration, so that every later delivery flows through automation.

15.3 Rolling Back Deployments

15.3.1 Why Roll Back

Hook. A new release goes live on Friday evening, and by Saturday morning the support inbox is full. Which instinct wins: "let me fix this live, right now" or "get everyone back on the old version first"? This section explains why the second instinct is the professional one.

The main situation where you need a rollback is when something goes wrong in the production environment. At that point it is essential to roll back to the previous version, because debugging a problem inside a running production environment is very difficult. If you try to debug live, it takes a long time, it results in late nights, and the result probably will not be fruitful. The right way to restore service to your users is to revert: if something goes wrong, go back to the previous version so that you can at least serve your customers and not face any penalty. Only afterwards do you debug the failure in the comfortable zone of your normal working hours, to identify where the problem was and how to fix it.

Formalize. A rollback is the act of reverting a deployment to a prior known-good version of the application. The professor's rule, in one line: restore service first, debug later. A rollback has a job — it is not a fix, it is a recovery. It returns users to the last version that worked, which stops the business damage (failed transactions, angry customers, penalty clauses) while the team investigates calmly. The counterpart idea is rolling forward: keeping the new version and shipping a new release with the error corrected. Rolling forward is essentially just another upgrade, so the lecture's focus is on rollback, the emergency path.

Think of a bus driver who takes the wrong turn on a night route. The professional move is not to stop the bus in traffic, open the engine, and try to rewire the navigation while passengers wait and the schedule burns; it is to return to the last known good stop and sort out the navigation afterwards, in the depot, during normal hours. The analogy breaks in one place: a bus can simply drive back, while software rollbacks can be complicated by data changes — which is why section 15.3.4 exists.

Worked decision — roll back or debug live? Suppose a payment portal serves 10,000 concurrent users and a new release breaks the payment confirmation screen.

Option	Time to restore service	Risk	Result for users
Debug live in production	Unknown — likely hours of late-night investigation while the site is broken	High: experiments in production can make things worse; the failure is happening in a stressed environment	Payment failure continues; revenue and trust bleed
Roll back to previous version	20 minutes (scripted re-deploy of the last good build)	Low: the previous version already served users for weeks	Service restored; payments work again

Sense-check: the rollback restores service in 20 minutes; the live debug has no promised end time and no guarantee. The professor's point: roll back, and do the debugging later, in working hours, on the broken release — not on the users.

Pitfalls — why teams fail to roll back.

Ego or pressure ("we just deployed it"). Teams often try to fix forward at 3 AM to justify the release. The professor's counter: the result of live debugging is usually late nights and no fruitful outcome.
No backup of data and file system. A rollback that cannot restore the database is a rollback that does not work (constraints in section 15.3.4).
An unpracticed rollback plan. The rollback path that has never been executed fails exactly when it is needed; practice is part of the plan.
Assuming rollback is instant. As the classroom stories below show, real rollbacks take minutes to hours; the release plan must expect that, not hope it away.

15.3.2 Methods of Rolling Back

There are several methods of performing a rollback. You can do it with blue-green deployment, and canary releasing can also be used to perform zero downtime releases as well as rollbacks. Both are covered in detail later in this document.

The three main paths. (1) Redeploy the previous good version: with an automated deployment process, the simplest rollback is to run the same script with the previous version — a fixed-time operation using the same process already tested hundreds of times. (2) Blue-green deployment: keep the previous version running on the unused environment, so rollback is just switching traffic back (section 15.5). (3) Canary releasing: if the new version sits on only a subset of instances, rollback is just rerouting users away from that subset (section 15.6). Method 1 works everywhere but causes a short downtime; methods 2 and 3 roll back almost instantaneously and so double as zero downtime rollbacks — the topic of section 15.4.

15.3.3 Student War Stories: Real Rollbacks in Production

The classroom produced several real rollback stories. Each one is a different confusion point — a different way a release can go wrong — so each keeps its own question-and-resolution flow.

Q: Can you mention any scenario or situation where you saw a rollback, or where you participated in a rollback in your team?

A: One participant described a deployment that disturbed previously working functionality in production, because a third-party integration to the application was not working on production. Another said that backward compatibility led to a rollback once in their project. On that, the instructor connected it to the previous session: we need to consider backward compatibility before we release the feature.

Worked war story 1 — the banking release (Standard Chartered, 2012). A participant from a banking company that processes crores of transactions shared this: if something goes wrong, the transaction gets reversed. They deployed code on a Friday for almost 17 countries. It was a major release with a logic change, and the calculation for each country was different — that calculation was missed in the logic. As a result, all the transactions for the users were reflecting in a different way than they should.

Trigger: a per-country calculation was left out of the new logic — the same code path ran for all 17 countries.
Impact: every user's transactions were computed wrongly; business impact huge, because money movement is involved.
Rollback method: the team used the version control system to rebase the old code from the new code — reverting the source to the previous version, rebuilding, and redeploying.
Time taken: it took almost five to six hours to change the code and set it back to the older version; after that, things worked fine.
Strategy at the time: no blue-green or canary; this was 2012, and the platform was not cloud — cloud itself was new to the industry. The instructor's note: banking makes life difficult in this area — banking applications are not really stable environments to work in.
Lesson: such a rollback is quite difficult because the business impact is huge; you have to be very careful, you need more validation on test coverage, and you need validations from the business side.

Sense-check: five to six hours of service degradation is a long recovery, which is exactly why later sections push blue-green and canary — both make rollback a traffic switch measured in minutes, not a source rebase measured in hours.

Q: (About a stock trading broker application) We used to give authorization and authentication with JWT tokens initially. The client then wanted to move to AWS Cognito, so we made all the port changes. When we deployed to production, we came to know that we had not provided the new token to the existing customers — the existing system was not able to do so. We could not test this in production or in the local environment either. That was the situation we faced about three months back.

A: In that situation we rolled back, and then we made all the changes: for the existing customers we made an entry manually in the database, and then we created a two-way authorization for the old and the new token flows. The instructor pointed out that this example connects to the previous session on backward compatibility: your DB schema should support the existing customer.

Follow-up Q: How much time did the rollback take for your team? I know the problem is small, but identifying the problem is the difficult part.

A: It was an Australian client, so the deployment happened very early in the morning. By around 7 AM there were a lot of mails — users were not able to log in. When we checked the login, we saw that no user could log in. So we suddenly made a rollback, identified the issue, rolled back, and fixed the issue. After one week we made another release with all the fixes. Almost one day was needed to reach the customer, because the big feature was moving everything to AWS and the staff network — security, payment, and all the branding.

Worked war story 2 — the Salesforce Service Cloud outage. One participant implemented Service Cloud Voice of Salesforce. One functionality was down in production, so they fixed the bug and deployed it. Their system is mostly dependent on APIs — most functionalities go down if the APIs are down. Customers have a self-service portal around Service Cloud where they log in and see their data, and that data also comes from the API. The moment the code was released to production, all the APIs went down and the whole production went down.

Trigger: a single important header was missing from an API request — "a slightly hilarious situation: we were trying to fix one thing and ended up making the whole production environment down."
Rollback problem: even using GitLab, one of the best version control tools, the team could not roll back — a caching issue: the pipeline's pointers were not catching up that header. Caching happens because the pointers point to a particular version, a particular snapshot, in GitLab.
Resolution: the code is on GitLab, but MuleSoft is used to deploy it. A previous release had modified the endpoints; when the pipeline was re-triggered on the incident release, the missing header surfaced. Eventually all changes were rolled back — and "everybody was laughing because the issue was really small."
Time taken: four to five hours to figure out the root cause and solution, with teams across the US (San Francisco, Texas) and India working through the night, some going to sleep at 5 AM, racing to restore service before business hours.

Sense-check: a one-header mistake took down an entire production environment, and the tooling (caching) delayed even the rollback — proof that the rollback path itself must be exercised, which is the lesson the professor draws in the constraints below.

Q: (From a Salesforce release manager) We have a lot of parallel releases: a major release always goes within a six months timeline, some minor releases go in parallel on a different timeline, and some enhancement releases go on a separate timeline — all going to the same production environment. There was a scenario where a few minor releases went to production and some part of them was missed being retrofitted in the major release part.

A: All major releases should be developed on top of whatever the latest code is in production, and then we do the delta deployment: if we are in release two and the release one code already went to production, we never test the release one code; whatever new commits are created as part of release two, we deploy those. In the scenario described, the code that was missed being retrofitted to the major release created an issue with the territory alignment functionality, and it messed up in production. We have a well-defined rollback strategy: in Salesforce, there is an XML file called destructiveChanges.xml, where we maintain all the metadata which we have to roll back. We just put all that metadata there and run the deployment; it runs as a rollback and rolls back within minutes only — it is instantaneous, it won't take much time.

Q: (About blue-green on two cloud platforms) I was managing a team, and it was a blue green deployment where we used to have two stacks, for Azure and AWS. We used to create stacks, and in those stacks we had all the applications and all the code, and we created an image for that; for that image, if something goes wrong, we used to revert back the things. Sometimes because of application dependency issues and some code issues, we used to revert back from newer to older ones — blue green deployment as a part of that.

A: Great — blue green deployment is on today's agenda, including what kind of constraints exist and what kind of considerations we should have.

Worked synthesis — what the war stories teach. Line up the five incidents side by side:

Story	Trigger	Rollback mechanism	Time to restore	Key lesson
Third-party integration / backward compatibility	A new deployment disturbed previously working functionality	(participant rollback)	—	Consider backward compatibility before releasing the feature
Standard Chartered banking (2012, 17 countries)	Per-country calculation missing from the new logic	Rebase old code from new code in version control	5–6 hours	Business impact is huge in banking; more test coverage and business-side validation needed
Stock broker JWT → AWS Cognito	Existing customers never given the new token	Roll back + manual database entry + two-way authorization (old and new token flows)	Same day; re-release after one week	DB schema must support the existing customer (backward compatibility)
Salesforce Service Cloud Voice	One header missing from an API request	Roll back via GitLab, blocked by a caching issue	4–5 hours	Even small mistakes cause big outages; the rollback tooling must be dependable
Salesforce parallel releases	Minor-release code missed in the major release (territory alignment)	destructiveChanges.xml metadata deployment	Minutes (instantaneous)	A defined rollback strategy makes recovery routine

Sense-check: every story follows the same shape — release, unexpected failure, restore the previous version, then fix. What differs is how long restoration took, and the teams with a defined rollback path (destructiveChanges.xml) recovered fastest.

Exam note: rollback questions connect to backward compatibility, which was covered in the previous session. Remember the professor's phrasing: your database schema should support the existing customer — and backward compatibility must be considered before the feature is released, not after the rollback is needed.

15.3.4 Constraints on Rolling Back

The first constraint is data. If your release process makes changes to your data, it becomes hard to roll back. There are two general principles to follow when creating a plan for rolling back a release:

Scope — when rollback gets hard. Rollback is cheap when the new version only changed code, and expensive when it changed state. Two situations make rollback especially difficult: (1) data changes — schema migrations, transformed records, or new data written by the new version, because reverting the code does not automatically revert the database; and (2) orchestrated releases — releases that span more than one system, because you must roll back every cooperating system in step, not just one application.

The two principles of a rollback plan.

Back up before the release. Ensure that the state of your production system — including databases and state held on the file system — is backed up before doing the release. Always back up the current state of the database as well as the file system. If the release changes data, the backup is the only trustworthy way back to the pre-release state.
Practice the rollback plan before every release. It should include restoring from the backup or migrating the database back, so that you make sure the plan works. The second practice is to actually migrate to that backed-up database and file system — an unexercised restore is a theory, not a plan.

Recap + bridge. Rollback is the emergency recovery path: restore service to the previous version first, debug later in working hours; keep data and file system backed up, and rehearse the rollback itself. The classroom stories show real rollbacks taking minutes to hours — which sets up the next concept: zero downtime release and rollback strategies that switch traffic in nearly instantaneous time, instead of rebasing code for five hours.

Real-world & domain connection. Rollbacks are a daily reality across the industry. Banking and trading systems (like the Standard Chartered story) treat releases with extreme caution because a bad calculation can reverse or misstate transactions worth crores; stock-broker platforms discovered that token migration (JWT to AWS Cognito) must never strand existing customers; Salesforce release managers institutionalized rollback with destructiveChanges.xml metadata deployment, making recovery a routine pipeline action. In the broader DevOps field, rollback capability is one of the metrics of a healthy release process: the fastest organizations restore production service in minutes, not days, precisely because they can revert to a previous known-good state on demand.

15.4 Zero Downtime Releases

15.4.1 Hot Deployment with Nearly Instantaneous Switches

Hook. What would a release look like if users could not tell it happened — no "scheduled maintenance" page, no service window, no pause? Zero downtime asks exactly that: can the switch from one version to the next be so fast that service never stops?

Zero downtime applies to the release as well as to the rollback. Zero downtime is a hot deployment: the process of switching users from one version of the application to the next version happens nearly instantaneously. And if you have to roll back, the rollback should happen within the same amount of time — nearly instantaneously as well. If something goes wrong, your rollback should not take a long time.

Formalize. A zero downtime release (also called hot deployment) is a release in which the process of switching users from one version of the application to the next happens nearly instantaneously — there is no window in which the service is unavailable. The defining pair of properties:

Instant switch in: users move from version $A$ to version $B$ with no service interruption.
Instant switch back: the rollback must happen within the same amount of time — if something goes wrong, your rollback should not take a long time.

The symmetry matters. A release technique is only "zero downtime" if both directions are fast; a strategy that releases instantly but rolls back in five hours (like the source-rebase rollback in section 15.3) is not zero downtime.

Visual intuition. Imagine a graph where the horizontal axis is time and the vertical axis is user-visible availability. A traditional release draws a deep notch: availability drops to zero for the maintenance window, then recovers. A zero-downtime release draws a flat line: at the release instant the line stays high, it merely changes color (version A traffic becomes version B traffic). The landmark is the switch instant — the point where the router, not the deployment, changes. One-sentence takeaway: in zero downtime, users never see the notch; they see only a different version serving them.

15.4.2 The Key Mantra: Decoupling

The key to a zero downtime release is decoupling the various parts, and that is where microservice applications play the major role. If you design your architecture in a microservice way, every service is decoupled: if something goes wrong in any one of the services, rollback is very easy, and even the zero downtime happens very easily. The key mantra for achieving zero downtime is decoupling your application — the various parts of the release process should happen independently as far as possible.

Intuition + analogy — the professor's mantra: decouple. Decoupling means designing the parts of a system so that each part can change (and fail, and roll back) without forcing the others to change with it. Think of a train: if every carriage were welded to the next, repairing one carriage stops the whole train. Couplers let a single carriage be swapped while the rest keeps rolling. A microservice application is a train of couplers: each service (an independently deployed unit that handles one business capability, e.g., "payments" or "search") can be released, rolled back, and scaled on its own. A monolith is the welded train — one change means one deployment of everything.

The analogy breaks in one place: a train's carriages carry separate cargo, but services usually talk to each other. Decoupling the deployment does not decouple the conversation — services still call each other's APIs, which is why the syllabus's three strategies (below) route traffic, not data, between versions.

What decoupling makes possible. The release process should be cut into independent movements:

Deploy separately from release. Deployment is installing a version into an environment; release is making that version visible to customers. When these are decoupled, code can sit in production while still switched off for users.
Upgrade shared resources ahead of the application. Databases, services, and static resources should be able to receive their new versions before the application that uses them switches over.
Roll back one service without touching its neighbors. A failing payments service can revert while search and catalog keep serving.

In microservice architectures this comes almost for free; in monoliths every release drags the whole system along, which is why the mantra matters.

Assumptions & scope. Zero downtime is not automatic — it depends on:

Two or more versions being able to run at once (blue-green needs two environments; canary needs a subset of instances; rolling upgrade tolerates mixed versions for a while). If the architecture cannot run two versions side by side, there is no instant switch.
Shared state being version-tolerant. The database is the classic breaking point: both versions read and write the same data, so schema changes must be additive and backward compatible (the "DB schema should support the existing customer" lesson from section 15.3).
Microservices helping, not guaranteeing. Decoupled services make zero downtime easier, but the technique still has to be designed per service.

15.4.3 The Three Strategies That Support It

Exam note: per the syllabus, three deployment strategies support zero downtime release and rollback: blue green deployment, canary releasing, and rolling upgrade — each covered in its own section below (15.5, 15.6, 15.7). A question from a previous class on what canary testing is gets answered in the canary section (15.6.3).

Comparison — the three strategies at a glance.

Dimension	Blue-green (15.5)	Canary (15.6)	Rolling upgrade (15.7)
Where the new version runs	A second, identical production environment	A subset of production instances	A few production instances at a time
How users switch	Traffic rerouted from green to blue (a single switch)	Small user set rerouted to the upgraded subset, then gradually more	Load balancer moves users to each upgraded instance as it is ready
Rollback speed	Nearly instantaneous (switch back)	Nearly instantaneous (reroute away from the subset)	Roll back only the upgraded instances
Resource cost	Highest — every resource duplicated	Moderate — subset + capacity overhead	Lowest — reuses existing instances
Best when	Budget allows full duplication	You want gradual exposure and real-user testing	Cost matters and mixed versions are acceptable

When to pick which: duplicate everything when budget permits (blue-green); expose gradually and test with real users when you want feedback before full rollout (canary); stay cheap and incremental when you accept mixed versions during the upgrade (rolling).

Recap + bridge. Zero downtime means the switch in and the switch back are nearly instantaneous; the key mantra is decoupling the parts of the release process, with microservices as the natural enabler. Three syllabus strategies deliver it — blue-green, canary, and rolling upgrade — and the next section starts with the most powerful of the three: blue-green deployment.

Real-world & domain connection. Zero downtime is the operating standard for internet-scale businesses: e-commerce sites cannot schedule "closed for upgrade" windows when customers span every time zone, and a release that costs five minutes of downtime costs real revenue. This is why the environment-based patterns in this section (blue-green, canary) let teams deploy during normal business hours instead of at midnight, and why companies like Facebook deploy the entire server fleet in about twenty minutes with no visible impact. In the wider DevOps field, zero-downtime capability is what separates "releases are projects with downtime windows" from "releases are routine, reversible traffic changes."

15.5 Blue-Green Deployment

15.5.1 Two Identical Production Environments

Hook. What if the release — and the rollback — were both just changing an address? Blue-green deployment turns releases into a traffic switch: the new version is already fully running before a single user is moved to it.

Blue green deployment is the most powerful technique for managing releases. You manage two identical versions of your production environment in parallel. For an application, say XYZ, you have a web server for the front-end code, an application server where you run the back-end services, and a database server to manage the database flow. Each of those instances has two identical slices — one green slice and one blue slice — so you are managing two identical production environments side by side.

Formalize. Blue-green deployment (also known as big flip or red-black deployment) is a release strategy in which two identical production environments — conventionally named blue and green — run side by side, and at any moment only one of them (the "live" one) receives user traffic. For application XYZ, both environments contain the full stack: a web server for the front-end code, an application server for the back-end services, and a database server. Each tier exists twice — a green slice and a blue slice — so the two environments are complete replicas of each other. The release is executed by changing which environment the users' requests are routed to; no application code changes are needed for the switch itself.

Intuition + analogy — the double kitchen. Think of a restaurant with two identical kitchens, side by side. Kitchen Green is serving tonight's customers. Kitchen Blue is the backup: it has identical equipment, identical ingredients, identical staff, fully running but empty. To "release" a new menu, the chef prepares everything in Kitchen Blue first — the new menu is cooked, plated, and tested there with zero impact on the diners. At opening time, the host simply seats the customers in Kitchen Blue instead of Green. If a dish is terrible, customers are seated back in Kitchen Green, which has been kept running. The analogy breaks where the professor warns it breaks: the two kitchens usually share a pantry — the database. You cannot instantly give each kitchen its own copy of every dish in the pantry, and that is the hardest part of blue-green (section 15.5.3).

15.5.2 How a Release and Rollback Happen

Worked example — the version upgrade walkthrough (version 1.0 → 1.1).

Setup: the green environment runs version 1.0 of application XYZ, and users are routed to the green environment's IP address to access it. The blue environment runs the same 1.0 stack, idle and in sync.

Release preparation. The organization wants to release version 1.1. The team deploys 1.1 on the blue environment in parallel, while green keeps serving all users on 1.0 — production traffic is never disturbed.
Testing. All testing happens on blue: smoke tests, then alpha and beta testing. Anything broken is fixed on blue before any user sees it.
The switch. Once everything is running smooth and alpha and beta testing are clear, the team reroutes the users to the blue environment: the IP address — the endpoint the application is accessed from — changes to point at blue. Nearly instantaneously, users are accessing version 1.1. The release took the time of a routing change, not the time of a deployment.
Rollback (if needed). If any bug or major issue gets reported by end users while they use 1.1, the team just reroutes the users again to the previous version, 1.0: the endpoint changes back to the green environment, and every user is back on 1.0. Meanwhile the team works on the blue environment to find the root cause, correct the problem, and restore the service. As soon as the problem is corrected, the address changes again and users reroute back to 1.1.

Sense-check: at every instant some environment was serving users — green (1.0), then blue (1.1), then green again during the incident — so both the release and the rollback are zero-downtime traffic switches, exactly as section 15.4 promised.

The instructor's caution — theory vs. reality. Explaining this theoretically is really easy, but in a real-time environment the challenging part is the technology stacks and the configurations. It is not always as instantaneous as it sounds — several people in the class shared examples where the rollback took four to five hours. It can be phased in any organization depending on the technology stack and the complexity of the configuration. The routing switch is nearly instant; the surrounding work — database migration, configuration drift, sessions, caching layers — is what stretches the timeline to hours.

Visual intuition. Picture the two environments as two identical columns of boxes (web server, application server, database server). The router sits above them like a switchboard. Only one column is lit (receiving the user arrows). The release is the router arm swinging from the green column to the blue column — the diagram's one-sentence takeaway is that nothing in the columns moves during the switch; only the router arm moves. Blue-green deployments of this kind are also described in the textbook as one of the "environment-based release patterns," which need little or no change to application code because the live/target choice lives in the routing layer.

15.5.3 The Database Problem and Two Solutions

It is usually not possible to switch over directly from the green database to the blue database the way you do for back-end services or web services. Even if you have taken a backup, migrating the data from one release to the next takes time, and if there are any schema changes it becomes difficult again. Organizations generally opt for one of two solutions:

Scope — why the database resists the switch. Code is easy to duplicate; data is not. While green and blue can each run their own copies of the application code, the live data is a single evolving resource. If the new version changed the database schema, the green and blue databases no longer match, and the instant switch over would carry users to a database that is missing data, or back to one that is missing the new structure.

Solution 1 — read-only mode before the switch over. Put the application into read-only mode shortly before the switch over, when the switch happens from blue to green or green to blue: freeze writes, copy the live database, restore it into the target environment, run the migration, then switch. This only works for applications where write operations are not usually accessed by the end users. If your application is write-oriented — updates happen frequently, and the application is designed around write operations — you cannot restrict your end customers from using those services, and read-only mode is not a suitable choice. And a rollback after writes have resumed risks losing transactions written to the new database, so the window must be handled carefully.

Solution 2 — migrate the database independently of the upgrade process. Design your application so that you can migrate the database independently of the upgrade process. The upgrade happens in parallel but independently, without any dependency on the upgrade process: you design the application so you can manage the database part independently. This is the same decoupling logic that leads towards the zero downtime release. In practice this means making only additive database changes — add new tables, columns, and fields, but never mutate or delete existing ones — so that both the old and the new application versions can read and write the same schema at the same time. This is often called the expand/contract pattern: expand the schema first, release the application, and only later contract (remove) the obsolete objects.

15.5.4 Cost Considerations

In terms of cost, blue green deployment is really costly: you are completely separating replicas of each environment. Two environments are complete replicas of each other, so every resource you have is duplicated — the cost becomes double instead of single. The best idea is to opt for blue green deployment when the project budget has no constraint on cost, or the project has a huge amount to spare.

The cheaper variant — one environment, two virtual machines. There is a more cost-effective way of managing a blue green setup: use a single production environment but create two different virtual machines out of it. You have two copies of the application running side by side on the same environment; each VM has its own resources — its own port, its own file system — but both are working on the same server. One VM is for production and one is for the release, so you keep the blue-green idea at a lower cost. The trade-off is that you are pushing more virtual machines onto the same server, so the physical environment becomes the shared bottleneck.

Option	Cost	Isolation	Bottleneck
Two complete replica environments	Double — every server, license, and resource is duplicated	Full: environments cannot interfere	None (independent capacity)
One environment, two VMs	Near-single — same physical servers	Partial: VMs share the physical machine	The physical environment: both slices compete for its CPU, memory, and disk

When to pick which: duplicate the environment when the budget has no constraint on cost or the project has a huge amount to spare; use the two-VM variant when cost matters and the shared physical capacity is known to be enough.

Pitfalls — blue-green mistakes seen in practice.

Forgetting that green must stay live. If the team switches to blue and immediately decommissions green's resources to "save cost," the instant rollback option disappears — green must be kept running (and in sync) during the supervisory period after the switch.
Treating the switch as always instant. The professor's caution stands: configurations, technology stacks, and database migration can turn the "one-second switch" into hours — phase and rehearse the switch itself.
Skipping the database problem. Switching web and application servers but not the database means the new version may be running against old data; the two solutions in 15.5.3 are part of the switch plan, not an afterthought.
Assuming blue-green works for write-heavy apps with read-only mode. Read-only switchover is only suitable when end users do not normally write; forcing it on a write-oriented application just breaks the service in a different way.

Recap + bridge. Blue-green deployment keeps two identical production environments, releases by rerouting users to the new one, and rolls back by rerouting them again — nearly instant in both directions, with the database as the one resource that resists the switch. Its cost (duplicated resources) is the main reason organizations consider the next strategy, canary releasing, which upgrades a subset of the existing production environment instead.

Real-world & domain connection. Blue-green is one of the most widely used release patterns in the industry, under several names — red-black at Netflix and big flip in some textbooks. Retailers have used it even for point-of-sale systems, staging new client software in an inactive state and letting store managers choose the moment of release. In cloud environments the "switch" is often changing a load balancer target or a DNS record between two availability zones, which is why major platforms expose blue-green deployment as a managed feature. In the DevOps field it is valued for what this section emphasized: deployments during normal business hours, simple changeovers, and a rollback that is a switch, not a five-hour source rebase.

15.6 Canary Releasing

15.6.1 The Subset Approach

Hook. Blue-green duplicates your whole production environment — what if you could get the same early warning without the double bill, by testing the new version inside the live environment, on a small slice?

Canary releasing involves rolling out a new version of the application on a subset of your production servers. Every production server has instances — there could be multiple instances in your production environment — so you take a subset of those instances and release the next version of the application on the subset. It works like a canary in a coal mine: it quickly uncovers any problem with the new version without impacting the majority of users.

Intuition + analogy — the canary in the coal mine (the professor's analogy). Coal miners carried caged canaries into tunnels: birds are far more sensitive to carbon monoxide than humans, so a dying canary warned the miners to evacuate before the gas reached deadly levels — a small, cheap casualty that protected the many. Canary releasing applies the same logic to software: a subset of production instances is upgraded first, and a small set of users is exposed to it, so problems with the new version are uncovered early without impacting the majority of users. Where the analogy breaks: miners never deliberately poisoned canaries, but in software the "canary" is a real part of your product — the small group of users on the new version experiences real bugs if the release is bad, which is why the group is kept deliberately small.

15.6.2 How a Canary Release Works Step by Step

Purpose. Canary releasing addresses the question blue-green cannot answer cheaply: is this new version actually good — under real production load, with real users? It gives fast, honest feedback on a new version while limiting exposure, and it serves as a zero-downtime release and rollback strategy (rerouting users away from a bad subset is the rollback).

Inputs & outputs. Inputs: the production environment with its instances (each instance being one running copy of the application, e.g., one server or container), the new version (call it X plus one \u2014 version $X + 1$ ), a router/load balancer that can steer users to specific instances, and a small set of test users. Outputs: either the full production environment gradually upgraded to $X + 1$ , or a rollback with only the test subset affected.

Steps.

Pick a subset. Out of the production instances, choose a few to upgrade (for example, 4 out of 10 instances). These instances are upgraded to version $X + 1$ ; the rest keep running version $X$ .
Route a small set of users to it. Once a version is upgraded, a small set of users — not the whole user base — gets rerouted to the new version of the application, the version which holds the new feature. This small set can be direct end users or third-party organization people performing canary testing. The logs are triggered, and this set of users uses the new feature of your application, version $X + 1$ .
Observe. Watch the logs and feedback from the test set. Issues raised by that small set are the early warning — the canary singing (or dying).
Roll back or continue. If any issues or concerns are raised by that small set of users, simply start rerouting the end users back to the previous version of your server instances. The debugging then happens on that set of servers, and you can restore and fix the issue. If everything is correct — if the $X + 1$ version has no bugs — the next step is to pick up another subset of instances and perform the upgrade on them.
Repeat until done. Likewise, slowly and steadily, the full production environment gets upgraded to the next version of your application.

Trace — a 10-instance fleet moves from version 7 to version 8. The production environment has 10 instances, each comfortably serving about 100 simultaneous users (total capacity 1,000 users).

Step	Action	Instances on v7	Instances on v8	Users on v8	State
Start	All users on version 7	10	0	0	Stable
1	Upgrade 2 instances to v8 (the canary subset)	8	2	0	Both versions running; no users moved yet
2	Reroute 100 users (internal testers + third-party canary testers) to the 2 v8 instances	8	2	100	Real production traffic on v8; logs flowing
3	No issues for 24 hours → upgrade 2 more instances, reroute 300 more users	6	4	400	Exposure grows in steps
4	Still healthy → upgrade 2 more, reroute 300 more	4	6	700	Majority of capacity now v8
5	Remaining 2 instances upgraded; all 1,000 users routed to v8	0	10	1000	Release complete

Rollback variant: suppose at step 3 a crash appears in v8's checkout flow. The team reroutes the 400 users on v8 back to the v7 instances, and the v8 instances are taken out of service for debugging. Only the exposed 400 users were ever affected; the 600 who stayed on v7 never noticed.

Sense-check: the release and the rollback are both just router actions on a subset — zero downtime for the unaffected majority, exactly as section 15.4 required.

15.6.3 Canary Testing and Third-Party Testers

Many organizations opt for canary testing and provide this testing part to third-party organizations. Most organizations do not perform canary testing with their own people: they opt for third-party people, who will use the application as an end user, and that is the best way of doing testing. You get genuine feedback, because those users are totally new to the application — they are not part of the organization that developed it.

Formalize — what canary testing is (answering the question a previous class asked). Canary testing is the act of testing a new version in the real production environment on a limited, controlled set of users and instances — as opposed to testing in a staging environment, which is production-like but not production. It is conceptually similar to a beta test: the software is real, the users are real, but the exposure is limited. This is why the textbook lists canary testing among the ways to verify a deployment under genuine production load — something no staging environment can fully reproduce. The professor's point: third-party users are the best testers because they are totally new to the application — they have no mental model of how it "should" work, so they exercise it the way real customers will, and their feedback is genuine rather than filtered through the assumptions of the team that built it.

15.6.4 Constraints: Capacity Impact and Data

Q: Can anybody tell me any challenge, problem, or constraint you see when you perform canary releasing?

A: The data constraint is everywhere — whether you talk about blue green deployment, canary releasing, or rolling upgrade, that constraint will be there. This is where people are opting to manage the database parts separately and independently. Another point raised was that more resources are needed: if your use case is not used by the canary user, but you actually own the product, you still carry that load. Region-related issues were also mentioned. One participant added that canary releasing is the best option for regional releases: say you want some feature changes for people in India — you can have those server instances pointing to version 1.1; for the USA, which has another kind of constraint, you can point those instances to version 1.0. But when it comes to more resources, as the earlier point said, this will impact the capacity of your production environment, and that is where you will need the resources.

Worked example — the capacity concern made concrete. Say the example uses four instances, and those four instances are being upgraded. In the time period where the version is getting upgraded, your application supports 1000 users — the capacity serves 1000 people at a time. While the four instances are being upgraded, those instances are not getting used, so in a scenario where 1000 users are simultaneously accessing the application, you see performance degradation, because it impacts the capacity of your production environment.

Work the numbers: 10 instances serve 1,000 users, so each instance carries about 100 users. During the upgrade window, 4 instances are taken out of service to be upgraded — only 6 instances remain, capacity drops to $6 \times 100 = 600$ users. If all 1,000 users try to log in simultaneously, 400 of them have no home: the remaining instances run overloaded, response times climb, and users experience degradation. In a blue green environment you had an identical environment with identical resources — that was the basic logic of blue green deployment: capacity was untouched because a full second environment absorbed the release work. In canary releasing you are directly upgrading the application on the production environment itself by creating a subset of instances, so that definitely impacts the capacity. That is the major concern.

Sense-check: the numbers show why canary is cheaper than blue-green but not free: every instance under upgrade is an instance not serving traffic, and the lost capacity must be absorbed by the remaining instances (or compensated with spare capacity) during the rollout.

15.6.5 Benefits: Alpha and Beta Testing, Feature Adoption, and Capacity Verification

Canary releasing is the best way of doing alpha and beta testing, because you can give those instances to a third-party vendor or organization that performs the canary testing — the best way of doing alpha beta testing of your application.

Some companies measure the usage of new features and kill them if not enough people use them. If you are not performing canary testing, but you want to see whether the application's new features are attracting end users, you reroute a few end users to the new version and observe whether they use it. If they are not using it, the new version does not generate any revenue, so you roll back to the previous version, and the organization makes the decision not to get upgraded to that particular version; instead the team has to rework the features in a way that will attract end users.

Finally, canary releasing is the best way of measuring or verifying whether the application is meeting its capacity requirement. You create a subset of instances and see whether the application sustains with the capacity — even in the situation where four instances were separated for the upgrade, you can check whether the application is sustaining the capacity. It is the best way of cross-verifying the capacity requirement by gradually ramping up the load: slowly rerouting more and more users to the application.

Exam note: canary releasing is the best way of doing alpha and beta testing and the best way of measuring or verifying capacity requirements — both are examinable conclusions of this section. The professor's related answers: the data constraint applies to all three strategies (blue-green, canary, rolling upgrade), and the capacity of the production environment is the major concern of canary releasing.

Pitfalls — canary mistakes.

Ignoring the capacity hit. Upgraded instances serve nobody during the upgrade; if total demand exceeds remaining capacity, users on the old version experience degradation even though nothing about them changed.
Letting the canary test with only happy paths. The whole point is genuine usage; if the "small set" is scripted or in-house, the feedback is not genuine.
Keeping too many versions alive. The textbook warns to keep as few versions in production as possible — supporting many versions simultaneously is painful; keep the number of canaries to a minimum.
Forgetting the data constraint. Both versions share the database; any shared resource must work with all versions in production, or the database part must be managed separately and independently (the same decoupling logic as section 15.5.3).

Recap + bridge. Canary releasing upgrades a subset of production instances, exposes a small set of users to version $X + 1$ , and rolls back or proceeds based on what those users reveal; it costs less than blue-green but consumes production capacity during the rollout. The next section removes even the subset: rolling upgrade, which replaces instances one at a time with almost no extra resources at all.

Real-world & domain connection. Canary releasing is standard practice at the largest scale: Facebook pushes to internal-employee servers first (its A1 group), then a small percentage of customers (A2), and only then the rest of the fleet (A3) — a staged canary rollout; Etsy and Netflix use the same idea with automated rollback when monitoring detects degradation. The capacity-verification benefit is heavily used in industry: when a production environment is too large to replicate for load testing, teams ramp up traffic on a canary subset and watch response times and resource metrics — the low-risk way to prove a new version can handle real load. Feature adoption measurement (killing features nobody uses) turns the canary from a release tool into a business experiment tool, deciding what ships based on usage, not opinion.

15.7 Rolling Upgrade

15.7.1 One Instance at a Time

Hook. Canary releasing protects the majority of users but takes instances out of service in groups. What if you could upgrade with almost no extra resources and almost no capacity loss — by changing one instance at a time?

Rolling upgrade is different from canary releasing: you are not creating a subset of instances on which you upgrade. Instead, it consists of deploying a small number of new version systems at a time directly to the production environment. You pick one instance at a time, upgrade that instance with the new version of your application, and slowly and steadily the full production environment gets updated. If something goes wrong, the instances where the new upgrade happened are rolled back to the previous version. You are not even rerouting the users — the production environment picks up an instance and upgrades it to the new release.

Purpose. Rolling upgrade is the zero-downtime strategy for teams that cannot afford blue-green's doubled resources and want less capacity disruption than canary's group upgrades. It replaces the production environment's instances one by one, keeping service continuous throughout.

Inputs & outputs. Inputs: the production environment (N running instances, all on the old version), the new version, and an orchestration mechanism — in the cloud, typically the platform's load balancer service (the professor names the AWS elastic load balancer, ELB). Outputs: all N instances running the new version, with the load balancer having removed each old instance from service before its replacement was registered.

Steps.

Take one instance out of rotation. The load balancer stops sending user requests to a single old-version instance (deregister it) — one instance out of N is barely felt by users.
Upgrade that instance. Replace its system with the new version; as soon as you upgrade an instance with the new version, you turn off the old version of the system on that instance.
Verify before proceeding. Before you remove the original system, make sure the new version system is serving the purpose — smoke-check that it handles traffic correctly.
Put it back into rotation. The load balancer registers the upgraded instance and starts routing users to it again. Slowly and steadily, with the load balancing, the application reroutes to those particular servers to serve the end users.
Repeat. Instead of creating a subset of instances, it picks up the instances of your production environment one by one and upgrades each to the next version; if everything is serving the purpose and everything is correct, it moves to the next instance.
Roll back per instance if needed. If there are any issues, you can track down whether they were related to the new upgrade, and if so, just roll back to the older version on the instances where the system got upgraded.

Trace — 10 instances roll from version 4 to version 5, one at a time. Each upgrade cycle takes 5 minutes (deregister, install, verify, register). The fleet of 10 instances serves 1,000 users (about 100 users per instance).

Cycle	Out of rotation	Upgraded to v5	Serving v4	Users affected	Total elapsed
Start	0	0	10	0	—
1	1 (instance 1)	0 → 1	9	0 (the 100 users are redistributed across the remaining 9)	5 min
2	1 (instance 2)	1 → 2	8	0	10 min
…	…	…	…	0	…
10	1 (instance 10)	9 → 10	0	0	50 min

Capacity check at the worst moment: 9 of 10 instances are serving during each cycle — the fleet temporarily carries 1,000 users on 900-user capacity, a 10% squeeze absorbed by the load balancer, versus the 40% squeeze of the four-instance canary in section 15.6. Sense-check: after 50 minutes all 10 instances run version 5, no user was ever without service, and no duplicate environment was ever provisioned — the defining trade of the rolling upgrade.

Assumptions & scope — mixed versions. During the rollout the environment deliberately runs two versions of the application at once (some instances on v4, some on v5). The textbook calls this out as the rolling upgrade's built-in risk: a mixed-version race condition can occur when a client's request lands on a v5 instance, receives version-5 state, and the client's next request lands on a v4 instance that does not understand that state — producing an error. Avoiding it requires either forward/backward compatibility between versions, feature toggles that keep new features off until all instances are upgraded, or version-aware routing. Rolling upgrade also assumes the upgrade of one instance is fast and safe; if each instance takes hours to migrate (for example, because of local data), the rollout duration becomes a real cost.

15.7.2 Cloud Support: AWS Elastic Load Balancer

You can think of the AWS elastic load balancer, which is the best feature of AWS that supports this rolling upgrade. What happens: as soon as you upgrade an instance with the new version, you turn off the old version of the system on that instance. Before you remove the original system, make sure the new version system is serving the purpose. Instead of creating a subset of instances, it picks up the instances of your production environment one by one and upgrades each to the next version; if everything is serving the purpose and everything is correct, it moves to the next instance. Slowly and steadily, with the load balancing, the application reroutes to those particular servers to serve the end users. If there are any issues, you can track down whether they were related to the new upgrade, and if so, just roll back to the older version on the instances where the system got upgraded.

Q: Is there a similar feature in Azure, like the AWS ELB, for rolling over?

A: Yes. Azure App Gateway provides the same kind of service. Every cloud platform provides the same kinds of services and features, so the same rolling upgrade pattern is available on other providers too.

Q: (About choosing between cloud platforms) How do you decide which one to opt for?

A: It is difficult to choose, but from personal experience: a lab was built to perform a whole CI/CD pipeline directly on the Azure platform by using Azure services, and working with Azure felt a little bit easy. Azure Pipelines is easier, and even with containerized apps it has become very easy and user friendly when it comes to Azure. For AWS, back-end services were deployed on EC2 instances and front-end web application code on an S3 bucket, with the S3 bucket connected to the EC2 instance, but connecting with Jenkins felt like a restricted way when working with AWS. Azure made it easier. Azure is also free in parts — it has some services which are open access, where you do not have to pay a single penny — which is why this was tried. Even connecting Azure with SonarQube worked: you can create the instances of SonarQube and connect them to Azure Pipelines; many of the services are freely available. A step-by-step tutorial document is planned so the whole pipeline on Azure can be shared with the team.

15.7.3 Benefits: Cost Effective and Risk Effective

The first benefit of rolling upgrade is that it is cost effective, because it is a cloud platform service: organizations generally go with the cloud platform, and the cloud provides this rolling upgrade service. Since the cloud is pay per use, it charges based on usage, so it is cost effective — you do not have to manage two independent servers. The second benefit is that it is risk effective: you are not hampering the capacity or the resources of your existing environment. These are the two major benefits of rolling upgrade.

Recap + bridge. Rolling upgrade replaces production instances one at a time through the platform's load balancer (AWS ELB, Azure App Gateway, and equivalents on every cloud), verifying each instance before moving on — cost effective because no duplicate environment is needed, and risk effective because capacity is barely touched. This completes the syllabus's trio of zero-downtime strategies; the next section turns from how to release safely to what to do when a defect is so urgent it tempts a team to skip the whole pipeline: emergency fixes.

Real-world & domain connection. Rolling upgrades are the default release behavior of modern container orchestrators (Kubernetes rolling updates, AWS ECS rolling deployments) and cloud load balancers, precisely because the platform handles deregistration, health checks, and registration automatically. The "verify before move on" step is implemented as a health check: the load balancer only promotes the instance once the new version passes its health probe, otherwise it rolls that instance back — the professor's rule in platform form. In the broader DevOps field, the trade-offs of the three strategies are part of standard release planning: blue-green when cost is no object and rollback speed matters most, canary when real-user validation matters, rolling when cost and capacity efficiency matter and the application can tolerate mixed versions briefly.

15.8 Emergency Fixes

15.8.1 What Counts as an Emergency Fix

Hook. A critical defect is found at 3 AM. The fastest-looking fix is to log into production and change a few lines right there. This section explains why that instinct — even in a genuine emergency — is almost always the wrong move.

An emergency fix is executed when there are issues in the latest upgrade of your application and those issues can be fixed with an emergency fix such as a patch fix, to make sure your services are up and active. It happens at the time of any issues, and DevOps suggests some best practices for such fixes.

Formalize. An emergency fix is a corrective change (typically a patch fix — a small, targeted correction to a deployed version) made because issues in the latest upgrade threaten to take services down or already have, and the goal is to make sure your services are up and active. The defining feature of an emergency is time pressure: normal fixes flow through the pipeline at a normal pace; an emergency fix is under pressure to restore service fast. Notice the professor's framing: it "happens at the time of any issues" — but the next subsection argues that most issues do not actually qualify as emergencies.

15.8.2 Best Practices for Emergency Fixes

The very first best practice: every emergency fix should run through the standard deployment pipeline. The organization should not bypass the deployment pipeline or directly do some fixes on the production environment; the suggested best practice is that emergency fixes should also follow the standard deployment pipeline.

The second best practice: one should always consider how many people are getting affected by this particular defect, how often it occurs, and how severe the defect is in terms of the impact on the end user. This consideration should be well checked before the team performs the emergency fix, because it usually impacts whether a fix is urgent: if the defect is not impacting the end user with higher likelihood or with higher impact, then one should not think of emergency fixes. It should be a normal modification — a normal bug fix in your application — that follows the normal process and does not end up being treated as an emergency fix.

Best practice 1 — never subvert the pipeline. Every emergency fix should run through the standard deployment pipeline: write the code, check it into version control, let the CI server trigger, build, analyze, test, and deploy through the environments. Bypassing the pipeline and fixing production directly has two destructive consequences: the change is not tested properly, which can produce regressions or patches that do not fix the problem and even make it worse; and the change is often not recorded, so the environment ends up in an unknown state that cannot be reproduced and breaks future deployments in unmanageable ways. The pipeline's speed is precisely what makes this possible: if the pipeline is fast, the "quick direct fix" saves no time worth the risk.

Best practice 2 — triage before you treat. Before performing an emergency fix, check three questions: how many people are affected by the defect, how often it occurs, and how severe its impact is on end users. This triage decides whether the fix is urgent at all: if the defect does not impact end users with high likelihood or high impact, it should be treated as a normal modification — a normal bug fix following the normal process — and not be escalated into an emergency fix. Treating every bug as an emergency is how teams end up with late-night patches and unrecorded changes.

Worked triage — is this an emergency? A release shipped with three known defects. Rate each one:

Defect	Who is affected	How often	Severity	Verdict
Payment confirmation shows the wrong currency symbol	All paying customers	Every transaction	High — money display is wrong	Emergency fix — high likelihood, high impact
Admin dashboard sorts dates in the wrong order	12 internal admins	Only when opening a specific report	Low	Normal bug fix — follow the normal process
Logout button missing on the mobile beta	200 beta testers	Occasionally	Low	Normal bug fix — beta is the environment for finding this

Sense-check: only the first defect meets the professor's bar ("higher likelihood or higher impact"); routing the other two through the pipeline as normal fixes keeps the emergency lane genuinely free for emergencies.

15.8.3 Dealing with a Defect Directly in Production

Sometimes teams end up dealing with a defect in the production environment itself, because of the business requirement or because of the customer need. In those situations, these are the considerations to think about:

Warning — the professor's production-defect rules (from the lecture).

Never do them late at night, and do not do it alone. You should pair somebody with yourself if you are dealing with a defect directly in the production environment. Fatigue and isolation are how small mistakes become bigger ones.
Make sure you have tested your emergency fix process. When you perform this emergency fix, it should be tested well before you release it to the production environment.
Only under extreme circumstances should you bypass the usual process. The usual process is: write the code, check it into your version control system, let your continuous integration server trigger, follow the automated build, automated code analysis, automated testing, then deploy into the delivery, deliver it into the staging and other environments, and finally deploy on the production environment. In an extreme circumstance you can fix the issue directly in the production environment with a package — release the package or do the changes there. Otherwise, even for emergency fixes, one should stick to the usual process.
Make sure you have tested making an emergency fix using your staging environment. Even if somebody is bypassing the usual process of releasing to the deployment environment, the solution of your emergency fix should be tested in a staging environment before it goes to reside in the production environment.
Sometimes it is always better to roll back to the previous version than to deploy a fix. Do some analysis to work out what the best solution is; preferably, one should roll back.

The decision ladder. When a defect strikes, work down this list in order: (1) roll back first — if the previous version restores service, that is usually the best answer, and the broken release can be debugged calmly afterwards (the lesson of section 15.3); (2) if rollback is impossible (data changed, orchestrated release), run an emergency fix through the pipeline — fast, tested, recorded; (3) only in extreme circumstances, and paired with a colleague in working hours, touch production directly — with the fix tested in staging first.

Pitfalls — how emergency fixing goes wrong.

The "quick fix" that is slower than the pipeline. Teams bypass automation to save minutes, then spend hours debugging the direct change that was never tested against the rest of the system.
Unrecorded changes. A direct fix that is not checked into version control disappears from the environment's history — the next deployment silently reverts it, or the next incident cannot be explained.
Patches that do not fix. Without the pipeline's automated tests, the "emergency" patch can miss the actual cause — the professor's phrase: it may not fix the problem and may even exacerbate it.
Solo, late-night heroics. The lecture's rule exists because tired, isolated debugging in production creates more incidents than it fixes.

Recap + bridge. Emergency fixes exist to keep services up and active, but they are still fixes: they follow the standard deployment pipeline, they pass the triage test (affected count, frequency, severity), and direct production work is the exception that comes with strict rules — never alone, never late at night, tested in staging first, and preferably replaced by a rollback. The final content section turns to the everyday craft of deployment: tips and tricks for building a deployment process that does not need emergencies in the first place.

Real-world & domain connection. The textbook's guidance matches the professor's exactly: "Do not, under any circumstances, subvert your process" — emergency fixes must go through the same build, deploy, test, and release process as any other change. In practice, teams that follow this see emergencies become routine: because the pipeline is fast (minutes, not weeks), a "hotfix" is just a small change that flows through the same checked, tested path as everything else, and the production environment stays reproducible. In the DevOps field this is part of the Second Way — feedback loops so fast that the emergency lane is rarely needed, and when it is, it is still an audited, tested lane.

15.9 Deployment Tips and Tricks

15.9.1 Let the People Who Deploy Create the Process

Hook. Who should write the deployment process — the developers who understand the code, or the operations people who actually run it? The professor's answer flips the common assumption: the deployers, because they know the deployment's real constraints.

The very first tip: the people who do the deployment should be involved in creating the deployment process, because those are the people who are well aware of the deployment process and know what the deployment constraints would be — they are the best suitable candidates to create it. This is where you see DevOps engineers as the current trend in the market: operation teams should be able to write their script by themselves. Operation people writing code — and that code is nothing but the deployment process. From the development teams, operation people will understand the constraints of the technology and the constraints of the configuration, and then operation people will be the ones who write this deployment process. Things go better when deployment and operations are friends.

Formalize — tip 1: build the process with the people who live it. The deployment process should be authored by (or co-authored with) the people who perform deployments. The reasoning is practical: they are the ones who know the deployment constraints — the technology limits, the configuration traps, the environment quirks — because they hit them every release. An operations team that can write its own deployment scripts is the modern DevOps engineer: their script is the deployment process, owned by the people who must trust it. The professor's one-line summary: things go better when deployment and operations are friends — when the teams cooperate instead of throwing artifacts over the wall.

15.9.2 Log Deployment Activities

Second: log the deployment activities. If your deployment process is not completely automated — including environment provisioning — it is important to log all the files that your automated deployment process copies or creates. The deployment activities should create logs so that you can audit the logs and find out the root cause of any issues, if any occur.

Formalize — tip 2: make every deployment leave a trace. Logging deployment activities means that the deployment process records what it did — which files it copied or created, which commands ran on which machines, when, and with what output — so that the log can later be audited to find the root cause of any issue. This is the operational half of the audit-log benefit from section 15.1: the pipeline's behavior becomes inspectable history. The professor's precise point: even when the deployment is not yet fully automated (for example, environment provisioning still involves steps), log every file your automated process copies or creates — the log is how you answer "what actually happened?" after an incident.

15.9.3 Don't Delete Old Files, Move Them

Third: do not delete the old files; rather move them, and rename them. You never know when you have to roll back, and what kind of old files will help you to switch over. Keeping the previous artifacts around makes the rollback path much easier.

Formalize — tip 3: archive, never destroy. When a deployment replaces old artifacts, do not delete the old files — move them aside and rename them (for example, keep the previous build's files in a versioned directory instead of overwriting them). You never know when you have to roll back, and you cannot predict which old file will help you switch over. Keeping the previous artifacts around makes the rollback path much easier — it is the file-system counterpart of the "keep green alive" rule in blue-green deployment (section 15.5): the old version is your emergency exit, so do not burn it.

15.9.4 Deployment Is the Whole Team's Responsibility

Fourth: deployment is the whole team's responsibility — a build and deployment expert is an anti-pattern. It is now the responsibility of every member. Even in a DevOps team, when you give the responsibility for one particular task to one team member, if there are any challenges or issues, the team is gelled together: the entire team should support the team member. Deployment is not one person's job. As said in earlier sessions, if any failures or issues happen, it is not the person who made the failure; rather, it is the team's responsibility to solve that failure, not to pinpoint the name of the person who created it. The team should know how to use the deployment script and maintain it, so that if that particular person is not there in the team, it does not mean deployment stops — other team members are still in a position to complete the deployment. There is no dependency on the people.

Pitfall — the "build and deployment expert" anti-pattern. A build and deployment expert is the single person (or single team) who "owns" deployment — and the professor names it an anti-pattern: a pattern of work that looks like a solution but creates more problems. The problems are concrete: when that person is unavailable, deployment stops; when that person makes a mistake, the failure is traced to an individual instead of being solved by the team; and the rest of the team never develops the skill. The alternative: deployment is the responsibility of every member — everyone knows how to use the deployment script and maintain it, and when one member is stuck, the team is gelled together: the entire team supports the member. There is no dependency on the people.

Recap + bridge — the four tips as a code of conduct. (1) The people who deploy create the process — they know its constraints. (2) Log every deployment activity — the log is the audit trail for root causes. (3) Move old files, never delete them — the old version is the rollback path. (4) Deployment is the whole team's job — the build-and-deployment expert is an anti-pattern. As a closing reference: contact sessions 11 and 12, referred from textbooks one and two, cover deployment and the deployment pipeline for further understanding.

Real-world & domain connection. These tips describe how modern release engineering actually runs. The logging tip is industry practice: deployment automation records automatically — for auditing and compliance purposes — which commands ran on which machines, when, who authorized them, and what the output was, and it makes deployment the only way to change an environment. The "operations writes the script" tip is the definition of the DevOps engineer role that dominates today's job market: operations people who code. And the team-responsibility tip mirrors the blameless culture of the DevOps movement — when releases fail, teams fix the process and the system, not the person. Together, the four tips are the everyday habits that make everything else in this lecture — human-free deployment, rehearsed pipelines, and quick rollbacks — actually work in a real organization.

Exam Guidance Summary

Exam note — the quiz structure under the current pedagogy. The course runs 16 contact sessions with four tutorial sessions, and every tutorial session is followed by a quiz. Quizzes were originally open for 24 hours only; on request this was extended to two days (48 hours). Each quiz takes about 20 minutes.

Of the four quizzes, the two best scores will be considered out of all four quizzes. If you failed to attempt quiz three, complete quiz four so you can score good marks. Quizzes cannot be reopened once the marks are uploaded, because reopening for only a few members would impact the members who already covered the quiz; it is a system dependency and system constraint.
One quiz question (the jar and var file item) contained a silly manual mistake while being embedded in the system; this will be accommodated for everyone at the time of counting the final score.
For the Git assignment: you may use Bitbucket instead of GitHub, but use Git Bash — the main aim of the assignment is to complete the Git commands. Submitting the step-by-step screenshots in a Word or PDF document is enough.

Coming up in the course. From the next contact session onwards, the topics will be continuous monitoring, configuration management, and current trends. Under current trends, the syllabus covers deploying to containerized solutions and automating the containerized configuration using Kubernetes, which is the last module of the course handout; Docker and ELK demos are also planned. The last tutorial session covered continuous integration with Jenkins and Maven; the next tutorial session will bring a Docker and Kubernetes demo, and the Jenkins master-slave configuration, which was not shown last time, will be covered.

Key Industry Applications

Version control systems as audit trails — GitHub and GitLab keep history so any release problem can be traced back to where it was triggered (the audit-log benefit of human-free deployment, section 15.1).
Jenkins with a Jenkinsfile — the classic example of an automated deployment script that runs the same way in every environment (the rehearsal benefit of section 15.2).
Legacy stacks show the contrast — Visual Source Safe (VSS) for version control plus BMC Remedy tickets and a separate implementation team made deployment a fully manual, slow process full of handshakes (section 15.1's student story).
AWS services used in the classroom stories — EC2 instances for back-end services, S3 buckets for front-end code, the Elastic Load Balancer for rolling upgrades (section 15.7), and Cognito for authentication — the JWT-to-Cognito migration was the source of a real rollback (section 15.3).
Azure equivalents — Azure Pipelines, Azure App Gateway for rolling upgrades, and free SonarQube integration for code analysis (section 15.7's cloud-platform discussion).
Salesforce release management — Service Cloud Voice, parallel major/minor/enhancement release timelines, and the destructiveChanges.xml file for fast metadata rollbacks; MuleSoft is used to deploy code from GitLab (section 15.3's war stories).
Banking and stock broker applications — Standard Chartered (2012) and stock broker apps where rollbacks carry huge business impact: transaction reversals, existing-customer token handling, and DB schema backward compatibility (sections 15.3 and 15.4).
Third-party canary testing — often outsourced so that genuinely new users exercise the application; some companies measure feature usage and kill features that do not attract enough users (section 15.6).
Microservice architecture — the practical enabler of zero downtime releases, because each service can be rolled back independently (section 15.4's decoupling mantra).

ITD Lecture 15 notes · Deployment Automation, Rollback, and Zero Downtime Strategies

Introduction to Devops· postgraduate· 2026-08-14

Sections Breakdown

115.1 Human-Free Deployment

Automating the release pipeline removes manual mistakes, makes delivery self-service for every team member, and rehearses the release path until it is routine, with every action traceable through audit logs.

215.2 Steps to Implement a Deployment Pipeline

The seven steps of pipeline implementation: value stream map and walking skeleton first, then automated build, deployment, unit tests, code analysis, acceptance tests, and release, all driven by the same script in every environment.

315.3 Rolling Back Deployments

Restore service first, debug later: why teams roll back, the three rollback methods, real classroom war stories from production, and the data constraints plus backup principles of a workable rollback plan.

415.4 Zero Downtime Releases

Zero downtime (hot deployment) switches users between versions nearly instantaneously in both directions; the key mantra is decoupling, with blue-green, canary, and rolling upgrade as the three supporting strategies.

515.5 Blue-Green Deployment

Two identical production environments with only one live at a time; releases and rollbacks become traffic switches, the database resists the switch and needs read-only mode or independent migration, and the cost doubles.

615.6 Canary Releasing

Upgrading a subset of production instances first, routing a small set of users (often third-party testers) to the new version, and rolling back by rerouting them away; capacity impact and shared data are the constraints.

715.7 Rolling Upgrade

Deploying a small number of new-version systems at a time, one instance at a time, through the platform's load balancer (AWS ELB, Azure App Gateway), with mixed-version risk and the benefits of cost and risk effectiveness.

815.8 Emergency Fixes

Best practices for emergency fixes: run them through the standard deployment pipeline, triage by affected count, frequency, and severity, and the strict rules for the rare direct production intervention.

915.9 Deployment Tips and Tricks

Four deployment tips: the people who deploy create the process, log every deployment activity, move old files instead of deleting them, and treat deployment as the whole team's responsibility.

10Exam Guidance Summary

Course logistics and exam intel: the two best quiz scores count, quiz timing and rules, the Git assignment details, and the topics coming up next.

11Key Industry Applications

Real-world anchors for the lecture's concepts: version control audit trails, Jenkinsfile pipelines, AWS and Azure services, Salesforce rollbacks, and microservices as the enabler of zero downtime.

Postgraduate students of software engineering and delivery

Exam Revision Notes

Below is the distilled, exam-ready core. Every entry comes from the full explanation above. Use this section for rapid review; return to the main notes when a point needs more context.

Human-Free Deployment

Must-know: Human-free deployment removes handoffs between people by automating build, test, analysis, and deployment; benefits are fewer manual errors (with audit-log traceability), self-service delivery for testers/sales, and a constantly rehearsed release path.

⚠️ Top pitfall: Assuming automation removes all errors: manual mistakes disappear, but design mistakes and environment drift remain, which is why audit logs and identical scripts across every environment matter.

Self-check: Why does the same deployment script running on every environment lower release risk?

Connects to: Section 15.2, Section 15.8

Steps to Implement a Deployment Pipeline

Must-know: The seven steps of pipeline implementation: value stream map + walking skeleton first, then automate build, deployment, unit tests, code analysis, acceptance tests, and the release process; one script drives every environment so the production release is the most-rehearsed step.

⚠️ Top pitfall: Automating the build but not the deployment, or letting environments drift so the 'same script' runs against different targets, silently invalidates the rehearsal benefit.

Self-check: Why does using the identical deployment script in staging and production make the pipeline safer, not just faster?

Connects to: Section 15.1, Section 15.3

Rolling Back Deployments

Must-know: On failure, roll back to the previous known-good version to restore service, then debug in working hours; rollback is constrained by data changes (back up database and file system before every release, practice the restore), and connects to backward compatibility from the previous session.

⚠️ Top pitfall: Debugging live in production instead of rolling back first — it causes late nights and rarely works; also unpracticed rollback plans and caches that block the rollback path fail exactly when needed.

Self-check: Why is 'your DB schema should support the existing customer' a backward compatibility statement, and which war story does it come from?

Connects to: Section 15.4, Section 15.5, Section 15.6, Section 15.8

Zero Downtime Releases

Must-know: Zero downtime = switching users between versions nearly instantaneously in BOTH directions (release and rollback); the key mantra is decoupling; the three syllabus strategies are blue-green deployment, canary releasing, and rolling upgrade.

⚠️ Top pitfall: Assuming zero downtime is automatic: it needs two versions runnable at once and version-tolerant shared state (especially the database); microservices make it easier but do not guarantee it.

Self-check: Why must the rollback be nearly instantaneous too, for a release to count as zero downtime?

Connects to: Section 15.5, Section 15.6, Section 15.7, Section 15.3

Blue-Green Deployment

Must-know: Blue-green: two identical production environments, only one live at a time; release = reroute users to the new environment; rollback = reroute back; database switching needs read-only mode (write-light apps) or independent/decoupled migration; cost is double because every resource is duplicated.

⚠️ Top pitfall: Believing the switch is always instantaneous — real rollbacks took four to five hours in class stories due to stacks, configurations, and databases; also decommissioning the old environment too early kills the instant rollback.

Self-check: Why is the database the hard part of a blue-green switch, and what are the two solutions the professor gives?

Connects to: Section 15.4, Section 15.6, Section 15.7

Canary Releasing

Must-know: Canary releasing: upgrade a subset of production instances, route a small user set to the new version, observe, then either reroute back (rollback) or upgrade more subsets until the whole fleet is on the new version; data constraint applies to all three strategies and capacity of the production environment is the major concern.

$available capacity = instances left \times \frac{1000 users}{10 instances}$

⚠️ Top pitfall: Overlooking the capacity hit during the upgrade window: upgraded instances serve no users, so simultaneous load (e.g., 1000 users with only 6 of 10 instances active) degrades performance for everyone.

Self-check: Why is canary releasing the best way to verify an application's capacity requirement?

Connects to: Section 15.4, Section 15.5, Section 15.7

Rolling Upgrade

Must-know: Rolling upgrade = deploy a small number of new-version systems at a time directly to production, one instance at a time, verified before moving on; supported by AWS Elastic Load Balancer and Azure App Gateway; benefits are cost effectiveness (pay-per-use cloud, no two servers) and risk effectiveness (no capacity/resource loss); rollback reverts only the upgraded instances.

⚠️ Top pitfall: Mixed-version problems during the rollout: with two versions serving simultaneously, a mixed-version race condition can error when a client's requests land on different versions; needs forward/backward compatibility or feature toggles.

Self-check: How does the AWS elastic load balancer perform a rolling upgrade, and what does 'make sure the new version system is serving the purpose' mean in practice?

Connects to: Section 15.4, Section 15.5, Section 15.6

Emergency Fixes

Must-know: Best practices for emergency fixes: (1) run every emergency fix through the standard deployment pipeline — never bypass it; (2) triage by how many people are affected, how often, and how severe; when dealing with a defect directly in production: never late at night, never alone (pair), test the fix process and the staging-tested fix, bypass the usual process only in extreme circumstances, and preferably roll back to the previous version instead of deploying a fix.

⚠️ Top pitfall: Bypassing the pipeline for a 'quick' production fix: the change goes untested (regressions, patches that do not fix), unrecorded (environment in an unknown, unreproducible state), and solo late-night heroics create more incidents.

Self-check: What three questions decide whether a defect deserves an emergency fix rather than a normal bug fix?

Connects to: Section 15.3, Section 15.1

Deployment Tips and Tricks

Must-know: The four deployment tips: (1) the people who do the deployment should create the deployment process — 'things go better when deployment and operations are friends'; (2) log deployment activities (files copied or created) for audit and root-cause finding; (3) never delete old files — move and rename them to keep the rollback path; (4) deployment is the whole team's responsibility — a build and deployment expert is an anti-pattern; contact sessions 11 and 12 (textbooks 1 and 2) cover deployment and the pipeline in depth.

⚠️ Top pitfall: The build-and-deployment expert anti-pattern: single-person ownership stops deployment when that person is absent, blames individuals instead of solving failures, and leaves the rest of the team without the skill.

Self-check: Why is a 'build and deployment expert' called an anti-pattern, and what replaces it?

Connects to: Section 15.1, Section 15.3, Section 15.5

Exam Guidance Summary

Must-know: Of the four quizzes the two best scores count; each quiz takes about 20 minutes with 48 hours to complete; quizzes cannot be reopened once marks are uploaded.

Self-check: How many of the four quizzes count toward the final score?

Key Industry Applications

Must-know: The industry applications that anchor each concept: audit-log version control, Jenkinsfile pipelines, AWS ELB rolling upgrades, Azure App Gateway, Salesforce destructiveChanges.xml rollbacks, and microservices enabling decoupled zero-downtime releases.

Self-check: Which cloud services implement rolling upgrades on AWS and Azure?

Connects to: Section 15.1, Section 15.3, Section 15.6, Section 15.7

Previous LectureLecture 14 · Maven and Jenkins: Building, Deploying, and Testing a Java Application

Next LectureLecture 16 · Continuous Monitoring and the ELK Stack

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

Prerequisite Knowledge

Previously Covered in This Subject

Deployment Automation, Rollback, and Zero Downtime Strategies

15.1 Human-Free Deployment

15.1.1 What Human-Free Deployment Means

15.1.2 The Release Process Gets Rehearsed, Tested, and Perfected

15.1.3 Student Questions and Answers

15.2 Steps to Implement a Deployment Pipeline

15.2.1 The Seven Steps

15.2.2 Why the Same Process Everywhere Lowers Risk

15.3 Rolling Back Deployments

15.3.1 Why Roll Back

15.3.2 Methods of Rolling Back

15.3.3 Student War Stories: Real Rollbacks in Production

15.3.4 Constraints on Rolling Back

15.4 Zero Downtime Releases

15.4.1 Hot Deployment with Nearly Instantaneous Switches

15.4.2 The Key Mantra: Decoupling

15.4.3 The Three Strategies That Support It

15.5 Blue-Green Deployment

15.5.1 Two Identical Production Environments

15.5.2 How a Release and Rollback Happen

15.5.3 The Database Problem and Two Solutions

15.5.4 Cost Considerations

15.6 Canary Releasing

15.6.1 The Subset Approach

15.6.2 How a Canary Release Works Step by Step

15.6.3 Canary Testing and Third-Party Testers

15.6.4 Constraints: Capacity Impact and Data

15.6.5 Benefits: Alpha and Beta Testing, Feature Adoption, and Capacity Verification

15.7 Rolling Upgrade

15.7.1 One Instance at a Time

15.7.2 Cloud Support: AWS Elastic Load Balancer

15.7.3 Benefits: Cost Effective and Risk Effective

15.8 Emergency Fixes

15.8.1 What Counts as an Emergency Fix

15.8.2 Best Practices for Emergency Fixes

15.8.3 Dealing with a Defect Directly in Production

15.9 Deployment Tips and Tricks

15.9.1 Let the People Who Deploy Create the Process

15.9.2 Log Deployment Activities

15.9.3 Don't Delete Old Files, Move Them

15.9.4 Deployment Is the Whole Team's Responsibility

Exam Guidance Summary

Key Industry Applications

ITD Lecture 15 notes · Deployment Automation, Rollback, and Zero Downtime Strategies

Sections Breakdown

Exam Revision Notes

Human-Free Deployment

Steps to Implement a Deployment Pipeline

Rolling Back Deployments

Zero Downtime Releases

Blue-Green Deployment

Canary Releasing

Rolling Upgrade

Emergency Fixes

Deployment Tips and Tricks

Exam Guidance Summary

Key Industry Applications

Was this lecture useful?

BitsNotes AI Assistant

Configure AI Chat

Sign in to use AI Chat