The Need for DevOps

3.1 What Is DevOps?

3.1.1 The Agenda and the Road Ahead

This session builds one argument in five steps, and each step leads into the next:

1. What DevOps is — a working definition you can apply to any project.
2. Why there is a need for DevOps — this starts with the problems that existed in software delivery and the principles every industry is expected to follow when delivering software. The problems explain the need.
3. How people adhere to DevOps practices inside an organization — the concrete day-to-day behaviors and categories of practice. The need explains the practices.
4. The continuous lifecycle of DevOps and how it evolved — how the practices reshape every phase of the software development life cycle (SDLC). The practices explain the lifecycle.
5. Case studies of who is adopting DevOps and how they started using the DevOps culture — real organizations that changed the way they work.

Keep the chain in mind while reading: problems → need → practices → lifecycle. Each link is the reason for the next, so the agenda is not five separate topics; it is one causal story about why software delivery became slow, and what DevOps does about it.

3.1.2 The Definition

Here is the definition that this course uses:

Two moments anchor the definition. The first is the commit: the moment a developer records a change (code, configuration, or anything else) into the shared system. The second is the moment that change is placed into normal production — running as part of the everyday service customers use. DevOps is about shrinking the gap between these two moments.

The definition also protects a second quality: it does not say "as fast as possible, no matter what." High quality is part of the target. A release that arrives quickly but breaks the service has failed the definition. Quality here means the deployed change is suitable for the people who use it — including end users, developers, and operations staff — covering things like availability, security, and reliability.

Notice what the definition does not do. It does not name any specific tool, and it does not force one methodology. If a practice is intended to shorten the time from commit to production while keeping quality high, it is a DevOps practice — whether it uses agile methods, automation tools, or plain forms of coordination. The definition is goal-oriented: the goal is the outcome, and the means are left open.

3.1.3 Reading the Definition: Practices, Tools, and People

The definition hands us two things to hold on to: practices and tools. But do not restrict the scope of DevOps to testing and development.

Why does the scope go beyond testing and development? Follow the two moments again. To reduce the time between commit and production, operations must be included in the collection of requirements — much earlier than the commit. And the goal does not end when the change reaches production: the deployed system must stay high quality throughout its life cycle, which pulls in monitoring and support after deployment. So DevOps reaches backward into requirements and forward into operations — testing and development are only the middle of the picture.

Keep the three-part mental model for the rest of the course: practices (the behaviors), tools (the automation that makes behaviors repeatable), and people (the collaboration that makes both work). A tool without a practice is shelfware; a practice without people willing to follow it is a poster.

3.1.4 The Three Groups in Software Delivery

To see why DevOps exists, look at the different people involved in building software. Three groups hold three perspectives.

Business people come up with the idea. Through demand management — an operational methodology for figuring out what the next demand in the market could be — they work out which idea to pursue. They carry high service-level expectations, because they want to strengthen the customer relationship: the customer should accept your service and stay, rather than move to some other industry.

The development team converts that idea into working software. They work at high speed. Through agile practices, they work in an incremental and iterative fashion, and every sprint of two, three, or four weeks delivers some working software. That keeps the speed intact.

The operations team delivers the software into the production environment. Their agenda is stability: the software must be stable enough to provide the expected services to the customer. Before working software goes into production, the operations team goes through a checklist — whether the command was run, whether the IP address was checked, and similar standard checklist items. They ask checklist-oriented questions of the development team. They need an approval — a stringent approval window — before pushing code into production. They check whether the development team adhered to the release notes, whether the audit logs were captured correctly, and whether the code is security compliant. One example of a compliance check: a password should not sit directly inside the code; it should be encrypted and kept somewhere outside the code scope. There could be other checks like it.

Each group is doing its job well by its own measure — and those measures pull in different directions. Speed wants change to move; stability wants change to be proven. That tension, not any one team's failure, is where the next problem comes from.

3.1.5 The Wall Between Development and Operations

Here is the problem those three groups create. Suppose sprint one is completed and handed over to the operations team to take into production. The development team has already moved to sprint two, to work on another feature. At that moment, the operations team asks questions about sprint one — checklist questions. The development team is busy, so responses to those mails are delayed.

Picture the scene as a diagram: two islands on either side of a wall. On the left, developers sprinting through features; on the right, operators holding a checklist, waiting for answers about the release that is already sitting on the wall. The mails pass back and forth over the top of the wall, each reply arriving late.

Even though there is speed, there is no sound, transparent collaboration between the two teams. A strong wall stands between them. Their responsibilities, aims, goals, and missions are different, even though they work on the same project. There is no single business KPI shared among all team members; everything is segregated — this is for development, this is for operations. And there is no trust, because there is no transparent collaboration.

This is where DevOps enters. DevOps exists to minimize the gap between development and operations, so that whenever a change is committed, that change resides in production. It increases collaboration among all team members. The wall does not vanish by goodwill alone; it comes down because practices, tools, and people are redesigned so both sides share the same pipeline and the same definition of done.

3.1.6 The Three Dimensions of DevOps

DevOps works on three dimensions at once:

1. The business-people aspect — about improving collaboration across the groups.
2. The tools-and-technology aspect — about getting the code from commit to production.
3. The best-practices aspect — because to implement those tools, you must adhere to best practices; a tool used badly is worse than no tool.

So DevOps comes with all three dimensions: collaboration, tools and technology, and best practices. Remove any one of them and the definition stops working: automation without collaboration rebuilds the wall faster; collaboration without automation keeps delivery slow; and practices without tools are not repeatable.

Where this matters in the real world: High-performing technology organizations treat commit-to-production time as a headline number. Companies like IMVU (a social entertainment service) deploy new code around 50 times a day; Facebook moved from weekly to multiple daily code pushes; Etsy performs dozens of routine deployments per day — all examples of the definition in action, where each committed change is tested, deployed, and monitored as a normal part of daily work. When you hear a company brag about "deployment frequency," you are hearing this definition, measured.

3.2 The Problems of Software Delivery

3.2.1 The Big Picture

The natural way to get to the "why" is to list the problems of software delivery and the principles every industry expects. Four problems stand out. They are not four disconnected annoyances — they form the failure pattern that DevOps was built to break:

1. Converting an idea into working software.
2. Getting reliable, rapid, low-risk releases.
3. Keeping ideal environments throughout the lifecycle.
4. Choosing the right software methodology for the problem statement.

The rest of this section walks through each problem, then generalizes all four into one lesson about finding the real bottleneck.

3.2.2 Converting an Idea into Working Software

The biggest problem in any software delivery is converting an idea into working software. The gap between "we have a good idea" and "customers use a product that embodies that idea" is where projects live and die.

To cross that gap, you need to understand the value stream map for your project: what steps to follow, what tasks your team has to perform, to convert that idea into a working product or service. Without that understanding, you are working in the dark. You cannot plan delivery, you cannot find waste, and you cannot tell which step is slowing everything down — because you do not even know what the steps are.

The value stream map is mentioned here as a problem-solving tool, and it gets its own full treatment later in this lecture (section 3.3) and again in the course when delivery and deployment strategy are discussed.

3.2.3 Reliable, Rapid, Low-Risk Releases

The second biggest problem is getting reliable, rapid, low-risk software releases. Releasing software into a production environment is always a hazard; there is a lot of chaos. Something can go wrong in the code, in the configuration, in the environment, or in the process itself — and on release day, you often cannot tell which one happened.

The industry evidence matches this worry. In a 2013 survey by XebiaLabs (a deployment tooling company), 7.5% of respondents reported their deployment process was "not reliable" and 57.5% said it "needs improvement"; nearly half named "too much inconsistency across environments and applications" as their biggest challenge. Real incidents also show the stakes: in August 2012, an upgrade failure at Knight Capital cost the firm 440 million dollars. Releases are a serious, expensive activity — and that is exactly why a good release process must be:

• Reliable — it works the same way every time, and when it fails, you know why.
• Rapid — the time to release stays short, so small changes go out in hours, not months.
• Low-risk — the process is rehearsed hundreds of times in lower environments, so the production release carries as little surprise as possible.

You want a release process that is reliable, that is rapid, and that carries as much low risk as possible. All three properties must hold together: a rapid process that is risky, or a reliable process that is slow, solves only half the problem.

3.2.4 Ideal Environments Throughout the Lifecycle

The third problem: we want the software to reach the production environment and, then, the other environments too — the staging environment, the pre-production environment, the capacity testing environment, the user acceptance testing (UAT) environment, and so on — at the same time.

An environment is the complete setting your application runs in: operating system, databases, middleware, network configuration, and all their settings — everything except the application code itself. The problem is that each environment must be kept in an ideal state — matching production closely enough that testing there means something.

All of these environments should stay ideal throughout the lifecycle of the project, so that testing is fulfilled very well, the team gains high confidence, and only then is the code pushed to production.

Why is this a "big problem"? Because in practice, environments drift. A patch is applied to one server and not another; a database setting is changed directly in production and never recorded; the staging environment slowly becomes a stranger to production. When that happens, tests pass in staging and the system fails in production — and nobody knows why, because the environments are not the same. Having ideal environments is a big problem whenever it comes to delivering software.

3.2.5 Choosing the Right Software Methodology

The fourth problem is choosing the right generic software methodology for the problem statement. Should you go with Agile? Within Agile, should you go with Scrum or XP (Extreme Programming)? Or should you use TDD (test-driven development), BDD (behavior-driven development), or FDD (feature-driven development)?

Each methodology suits a kind of problem statement. The table below gives the basic flavor of each:

If you pick a methodology that is not meant for such a problem statement, you will not get a fruitful result for your project. So before you start, understand the value stream map very well, and focus more on requirement gathering.

3.2.6 The Water Tank Analogy: Finding the Real Bottleneck

To generalize the problems of software delivery, here is a small real-time scenario. Imagine a water tank collecting water. Water flows through four different layers — four pipelines, four levels of filtration — and then resides in the tank. The speed of collecting the water is slow, and you would like to optimize it, to make it fast.

If you do not understand your core levels, you will probably go and do trial and error. Suppose you increase this gap, the gap at one filtration level — will it make the collection of water faster? Definitely not, because that is not the bottleneck. The bottleneck is somewhere else. Even though you are doing everything correctly — all your practices and processes are right — you are doing it at the wrong point, and that will not help. So you find the actual bottleneck and increase the gap there, to make the speed faster.

Once you did that, the flow improves. But next time, the speed is still not what you expected. So you need to improve another level. Will increasing the gap there help? No — the next bottleneck has moved. You locate the next bottleneck and optimize that, and so on, level by level.

The lesson generalizes directly: in software delivery, you must know your pipeline well enough to find the real constraint. Correct practices applied at the wrong point produce nothing. In a software pipeline, the "layers" are the steps of delivery — requirements, design, development, testing, deployment, operations — and the "capacity" is how much work each step can handle. If testing capacity is the bottleneck, speeding up development does not speed up delivery; the work piles up at testing instead. The water tank story is also why value stream maps matter — you cannot optimize a pipeline you cannot see.

3.2.7 Student Questions and Answers

A second question from the room was about hands-on practice — whether the environments discussed here are something the students themselves will build.

3.3 Value Stream Maps

3.3.1 What a Value Stream Map Is

A value stream map (VSM) is the path you follow to convert a requirement — once collected — into a working product. In normal terms: every step you adhere to, every process you follow, from the collected requirement to the working product, is your value stream map.

The idea comes from lean thinking: in any flow of work there is a sequence of activities that add value (the customer would pay for them) and other activities that do not (waiting, handoffs, rework, unnecessary approvals). The map makes the whole sequence visible — not just the activities, but also how long each takes and how much waiting happens between them. In lean terms, the value stream is "the sequence of activities required to design, produce, and deliver a good or service to a customer." For software, the value stream converts a business idea into a working service that delivers value to the customer.

A good value stream map names every step, the team that performs it, and two times for each step:

• Lead time — the total time from when the request is made until it is fulfilled, including waiting in queues.
• Process time — only the time someone is actually working on it.

The gap between these two is the waste hiding in your pipeline. A step may take 10 minutes of work but 3 days of queue time — and the map shows that.

3.3.2 A Worked Example: The E-Grocery Web Page

Suppose you have a requirement to implement a web page for your e-grocery application. What could your value stream map look like?

Even inside those broader tasks there are smaller tasks. When you reach the coding level, you have certain smaller tasks: creating a text box, creating a text box for the password, assigning it to a particular developer or one engineer. All of these steps, taken together, are your value stream map. The map is not one layer deep — each box contains its own sub-flow, and the sub-tasks are part of the same value stream.

3.3.3 Why Do the Value Stream Mapping Exercise

We do the value stream mapping exercise to remove as much waste as we can.

Compare with the water tank: there, the waste was small — the collection was slow because of a bottleneck. We removed that waste by increasing the gap, and made the flow valuable. In software, you want a valuable product to come out, rather than wasting your time on manual tasks or on stringent approval systems. You do not want that waste in your cycle. You want to achieve output faster and increase the value time in your value stream map.

The exercise has a practical track record. At Nordstrom (a large US retailer), a value stream mapping workshop on a slow, decades-old application found the real problem was not the technology at all: the request form asked store managers for an employee number they did not have, so forms bounced back and forth. Deleting that field cut processing time by four days; replacing a back-office PC with an on-floor tablet cut it further to seconds. Nobody would have found that waste without mapping the stream first.

The deeper methods — the ways people perform value stream mapping, and what lead time and cycle time mean in detail — come later, when the discussion reaches delivery and deployment strategy. For now, hold three facts: a value stream map is the path from collected requirement to working product; it exists to expose waste; and the biggest waste is usually waiting, not working.

3.3.4 Student Questions and Answers

A follow-up question asked for the definition in one sentence, with the example attached.

3.4 Common Release Anti-Patterns

3.4.1 Manual Deployment

An anti-pattern is a way of working that looks normal, is repeated widely, and reliably produces bad results. In the traditional world, deployment was a manual process, and manual deployment is the first and deepest anti-pattern. If you deploy manually, you face several challenges.

Signs of the anti-pattern. You will have extensive and detailed documentation. Your document will contain everything: open the command line editor, run this particular command, get the file from here, do the file transfer, make those files reside in the production environment, set the path, set the environment variables. These are a few of the manual steps of delivering to production or to any other environment. You rely on and strictly follow manual testing. Then you make frequent calls to the development team to understand certain aspects: why does this code have this particular IP address, from where should the DB access be performed, how do I connect to the DB. You depend on the development team for the explanation. You identify gaps and make frequent corrections in your release process.

Why it is dangerous. If something goes wrong — even when you followed every process — a manual mistake or a step you missed in the document creates the problem. Sometimes you sit with blurry eyes in front of a monitor until 2 a.m. to understand where the problem is, and the problem turns out to be very easy, very silly. The documentation is also always slightly out of date, because maintaining a deployment manual by hand is a job in itself; and a manual process cannot be tested cheaply — the only way to test it is to do it.

The remedy (what the anti-pattern is replaced with). Deployments should tend toward being fully automated. In the ideal, a human does two things: picks the version and the environment, and presses the "deploy" button. Everything else — file transfer, path setting, environment variables, configuration, restarts, smoke checks — is executed by a script that is itself checked into version control. The script is the documentation, and it is always up to date: if the script is wrong, the deployment simply will not work, and you fix the script, not a wiki page.

3.4.2 Deploying Only After Development Is Complete

Another anti-pattern: deploying to a production-like environment only after your development is complete. This was the traditional way of working, and some organizations still adhere to it — they do not push the code to the production environment until development is complete, even for small changes.

That is why CI/CD pipelines have two variations. CI (continuous integration) and CD (continuous delivery/deployment) describe pipelines that move every committed change forward automatically:

• Variation 1: continuously get the change — fetch the change from the git repository — and push it all the way until the production environment.
• Variation 2: fetch the code from the git repository and push until the staging environment; the push from staging to production afterwards is manual.

People choose the second way because they wait until development is complete; they do not want to push small changes into production. That varies from organization to organization, and it can be completely fine. But it is still an anti-pattern: it is not a good practice to restrict yourself to "until my code is completed I will not push that code to production."

Why does waiting hurt? Every week of waiting is a week in which the code and the production environment are not tested together. When the deployment finally happens, the batch of changes is large, and if anything fails, it is hard to know which change caused it. Deploying early and often — even to staging — keeps the delta small, so each deployment is a small, low-risk step instead of a big, high-risk leap.

3.4.3 Testing on Development Machines

As soon as the tester tests the system on the development machine, that is a challenge — it is not the correct way of working. Testing should be followed on a subsequent testing environment, not on development machines.

A development machine has the developer's local settings, local databases, local file layouts — a private world that production does not resemble. Tests that pass there prove only that the code works on that machine. The environment where the code will actually run — with its real configuration, real infrastructure versions, real constraints — is a different world, and it is the one that must be verified.

If you follow the traditional approach, releasing into staging is the first time the operation people interact with your new release. Until then, the operations team does not know what is happening in the code base. The code gets assembled by the development team; the operations team does not know what components exist, how the components are correlated, or even where the source code is — until the release reaches the staging environment and the developer says "we are done; here are the testing logs, the audits, the checklist; everything is done; now go and push this to production."

The result: no collaboration and no transparency between the two teams. Since there is no collaboration, the operations team feels the development team has no work — they just copy-paste and randomly write code. At the same time, the development team thinks the operations team has no work — they just push the code that was already written. This is how trust breaks between team members, and it impacts the overall software delivery.

3.4.4 Manual Configuration Management

The next anti-pattern is manual configuration management of the production environment. Manual configuration is very dependent: it creates a one-man dependency. If that person is not there, you will see a lot of challenges. The person gains more and more dependency in the team — "I am dependent, you cannot remove me." It is a kind of silo, and the DevOps culture does not want silos; it wants to break them.

Manual configuration brings specific consequences. Deployment differs between staging and production: whatever software exists in the production environment will not automatically be present in the staging environment; every environment has different configurations. If you do it manually, think about the scenario: if you have five environments, you keep five different documents with all the configuration information, and then manually check whether everything is at the same version — say it is Windows, what is the Windows patch level? If it is not the same, change it to the correct one.

The failure modes pile up:

• Lost documents — if by mistake any document is deleted, you are gone. The knowledge of that environment disappears with it.
• Different hosts behave differently — two servers in the same cluster end up with different patch levels, different connection pool settings, different library versions, and nobody recorded the difference.
• Slow environment preparation — it takes a long time to prepare an environment manually, because you sit, follow the document, and stay focused enough to minimize mistakes.
• No rollback — you cannot step back to the earlier configuration of your system if something goes wrong. "Fix forward" becomes the only option, and each fix-forward change that is not recorded makes the next problem worse.
• Direct modification — modifying the configuration directly on production systems was the practice people followed — and it is absolutely incorrect practice.

The remedy: all configuration for every environment — operating system patch levels, middleware settings, application configuration — is applied from version control through an automated process. Then five environments are no longer five documents; they are one script run five times, and the output is identical. You can also rebuild rather than repair: when a server drifts from the known-good state, you destroy it and recreate it from the script — what the industry calls treating servers like cattle, not pets.

3.5 Principles of Software Delivery

3.5.1 What Every Industry Wants

Every industry looks for the same things when releasing software. These principles are the positive mirror of the anti-patterns: where section 3.4 said "stop doing manual deployment," these principles say "here is what to do instead." Hold them together — an anti-pattern is the violation of a principle, and each principle is the remedy for an anti-pattern.

The six principles:

1. Repeatable, reliable, and automated.
2. Everything in version control.
3. Bring the pain forward.
4. Build quality in.
5. Done means released to production.
6. Shared responsibility and continuous improvement.

3.5.2 Repeatable, Reliable, and Automated

Industries want a repeatable, reliable process for releasing software. No manual work: they want a process that is repeatable and reliable. Automate almost everything — whatever you can do, try to automate it. That is the principle.

Why "repeatable"? A process you can repeat is a process you can rehearse. If the same script deploys to development, testing, staging, and production, then by the time a release reaches production, that exact deployment has already succeeded many times earlier in the pipeline. The production release stops being a first-time event and becomes routine. A manual process, by contrast, is never exactly the same twice — and different executions of a manual process are where "it worked yesterday" failures come from.

Why "reliable"? Reliability is what repeatability buys you: when the process is automated, the only question is whether the script and the environment are correct, not whether the operator remembered step 23. And automation makes the process auditable: an automated deployment records exactly what was run, on which machines, and what the result was.

3.5.3 Everything in Version Control

Keep everything in a version control system. Why? If something goes wrong, or any data gets misplaced, you can extract the previous version through your version control system. You can go back to the previous version of your codebase.

The principle goes beyond source code. If the environment is not in version control, then "go back to the previous version" is impossible for the very things that most often cause production failures: configuration files, patch levels, deployment scripts, database schemas. The industry finding is blunt: whether operations teams use version control predicts IT performance even better than whether development teams do — because there are far more configurable settings in an environment than in code, so that is where the drift hides.

Version control is the safety net under everything: any misplaced data, any bad change, any lost configuration — the previous good version is one command away.

3.5.4 Bring the Pain Forward

If it hurts, do it more frequently. Bring the pain forward rather than touching the riskiest part at the end. Pick the most risky or the most complicated part first — because with that part you will get more understanding about your product. Do it frequently and early, not late.

This principle inverts the natural instinct. The natural instinct is to postpone the painful, risky work — the tricky integration, the unfamiliar component, the awkward deployment — until "later, when things calm down." But the pain does not shrink with delay; it grows. Every week the risky part waits, the code base changes around it, and the moment it is finally touched, it breaks against a target that has moved.

Doing it early and frequently has a compounding payoff: each attempt teaches you something about the product and the process, the pain becomes familiar, and by the time the "big day" arrives, the risky part has already been done many times. (This is exactly what the release process does for deployments: the deployment to production is rehearsed dozens of times in lower environments before it is ever the real thing.)

3.5.5 Build Quality In

Build quality in. With respect to software delivery, build quality by catching defects as early as possible. If you catch the defect in an early phase, your scope is small, and the cost to fix that defect or fault is much lower — because the scope is small.

The logic runs on two axes:

• Scope: a defect caught during requirement analysis involves a document, a discussion, and a decision. The same defect caught after release involves code changes, re-testing, redeployment, customer communication, and support calls.
• Cost: industry studies of software projects consistently find that fixing a defect costs roughly ten times more at each later phase — a defect that costs 1x to fix at requirements time costs on the order of 10x at coding, 100x at release, and more once customers have it.

That is why the principle is "build quality in," not "inspect quality at the end": quality is cheapest when it is designed in from the start and verified continuously along the way — through automated tests that run on every change, not through a frantic final testing phase.

3.5.6 Done Means Released to Production

When the team says "done," done means the change should be released into a production environment. That is the meaning of done. Not "code is written," not "tests pass" — released to production.

This principle changes the team's vocabulary. "I finished the feature" is not a claim about a laptop; it is a claim about a running production system. The natural objection — "but we do not want to push this small change yet" — is exactly the anti-pattern from section 3.4.2. The definition of done forces the small, frequent, low-risk releases that the whole delivery pipeline is built to support.

3.5.7 Shared Responsibility and Continuous Improvement

Everybody is responsible for the delivery process; no single role owns the responsibility. Note the distinction: accountability is different, responsibility is different.

Responsibility is shared: everyone who touches the pipeline — business, development, testing, operations — owns the outcome of delivery and the quality of the process. When delivery is slow or broken, it is a team problem, not a ticket for one department. Accountability is a point of authority: one person or role can be called to answer for a specific outcome — the person who approved the release, the person who owns the service. Both concepts have their place; the principle says responsibility must be shared, while accountability can remain assigned.

And continuous improvement: we want to improve the process we are adhering to, continuously. The delivery process is itself a product: at the end of every iteration, the team reviews what slowed delivery down, and the next iteration starts from an improved process. This is the loop that turns a good pipeline into a fast pipeline over time — and it is why "done means released to production" does not end the work, it starts the next improvement cycle.

3.6 The Five Categories of DevOps Practices

3.6.1 Treat Operations as a First-Class Stakeholder

The first category: treat the operations members — the ops team — as first-class citizens from the point of view of requirements. The day you start your project, involve your operation team. You will start with requirement analysis — treat the operations people as a first-class stakeholder in that requirement analysis space.

"First-class" means they are not consulted as an afterthought; they sit at the requirements table from the first day, with the same standing as the business and the developers. Their requirements are real requirements: log messages must be understandable by an operator, monitoring hooks must exist, the service must be deployable and recoverable in the environment it will actually run in.

Why do we want to involve the operation team this early?

• To avoid miscommunication — the handoff between development and operations is where meanings get lost.
• To avoid rework — design decisions that ignore operations get redone later, at the most expensive possible moment.
• To be aware of the process and check dependencies, including dependencies in terms of the environment — what the software needs to run, and what the environment needs from the software.
• To get better support and clarity of support — a team that built the product with operations understands how to support it.
• Because operations is responsible for actual deployment — involvement has to happen at an early stage.

3.6.2 Make Development Responsible for Incident Handling

The second category: make the development team more responsible for relevant incident handling. Why? The development team should understand the workings of the operations team.

Once they understand what kind of incident could happen because of what kind of code practice they follow, those kinds of incidents will not come in the future. If developers know that a missing timeout will surface as a production outage at 3 a.m., they write the timeout. If they know that an unlogged error becomes an un-diagnosable incident, they add the logging. The knowledge of operations becomes a design constraint for development.

The practice is concrete: for a period after a new deployment, development carries the primary responsibility for incidents in that deployment — monitoring it, diagnosing it, fixing it or rolling it back. Only later does primary responsibility move to operations. This does two things at once: it forces the team that wrote the code to feel the consequences of how it behaves in production, and it brings collaboration between development and operations, because both now share the same incident loop.

3.6.3 Enforce One Development Process for Everyone

The third category: enforce the development process used by all — even your development team and your operational personnel should follow the same development process.

If developers use version control, agile rituals, code review, and automated tests, then operations scripts must live under the same rules: version control, review, tests, and the same pipeline. If operations staff perform ad-hoc manual steps that the rest of the team would never ship to production, those steps are where errors hide.

Why does uniformity matter? First, quality: deployment scripts are code, and code that is not reviewed and versioned will rot. Second, diagnosis: when every change — application or environment — flows through the same process, the history of "what changed" is one searchable story instead of a mystery. And third, trust: a shared process is a shared vocabulary; both sides can see what the other actually does.

3.6.4 Automate Application Deployment Continuously

The fourth category: continuous application deployment. Bring application deployment continuously, with as much automation as possible.

Every change committed to version control is automatically built, tested, and moved along the pipeline — and the pipeline itself is automated to the maximum extent. Two variations exist in practice, as noted in section 3.4.2: automatic flow all the way to production, or automatic flow to staging with a manual push from staging to production. Which variation an organization picks is its own decision; the category itself is about making the deployment continuous — meaning small, frequent, automated — rather than a rare, manual, big-bang event.

3.6.5 Develop Infrastructure as Code

The fifth category: develop infrastructure as code. Have deployment scripts; configuration as code. No more manual configuration management. Achieve automation with certain tools and technology.

Infrastructure as code means the environment is built by executing code, exactly as the application is built by compiling code: the same practices, the same version control, the same review. The script that creates a server, installs the operating system patches, sets the application configuration, and joins the load balancer is treated as a first-class artifact — because a misconfiguration in a deployment script is a defect just like a bug in application code, and it deserves the same discipline.

3.6.6 Student Questions and Answers

The second question asked whether the architecture-rework story was a real experience or a hypothetical.

3.7 The DevOps Lifecycle: Changes Across the SDLC

3.7.1 The Value Chain

The DevOps lifecycle process is a value chain — the one shown in the textbook. A value chain is the sequence of activities that jointly create value, from the idea to the running service; the textbook draws the DevOps processes in this notation, with each activity a step in the chain.

To see what DevOps actually adds, look at what changes at every phase of software project management, of the software development life cycle (SDLC). The seven phases below are the standard chain; for each phase, the change is the same shape: remove the handoff, add automation, involve the right people earlier.

3.7.2 Requirement Analysis

In requirement analysis, DevOps suggests treating the operations team as a first-class stakeholder, and getting their input when developing requirements. You need to be clear with their inputs.

This is category one of section 3.6 applied at its earliest possible point. The requirement document should include operations' needs: what log messages must look like, how the service will be monitored, what the environment dependencies are. A requirement set that only describes customer-facing behavior is incomplete — it describes the feature but not its life in production.

3.7.3 Development

In development: have a small team, with limited coordinations. No more people dependency — whenever you need anything, you should be clear with your process and easily get it through your process. And perform unit testing — unit testing should be done by the development team.

Two changes happen here. First, the team shrinks and self-contains: a small team decides quickly, and limited coordination means the team does not depend on other departments to make progress on its own work. Second, quality moves to the source: developers write and run unit tests as part of daily work, rather than handing code to a separate testing department and waiting weeks for the verdict.

3.7.4 Build

In the build phase: adhere to build tools to make the build process automated, and that build process should support continuous integration.

A build is the conversion of source code into runnable artifacts (binaries, installers, packages). The DevOps change: the build is triggered automatically by every commit, it runs in a dedicated environment rather than on a developer's machine, and its output is used everywhere downstream — the same artifact that passes tests in staging is the artifact that goes to production. This "build once, deploy everywhere" discipline is what makes the rest of the chain trustworthy.

3.7.5 Testing

In the testing phase: automated testing. The user acceptance testing should be completed, with automation tools and technology.

The change: tests run automatically on every change, in environments as close to production as possible. User acceptance testing (UAT) — where business users confirm the system delivers what they asked for — is still part of the chain, but it is supported by automation rather than performed entirely by hand. The feedback loop shortens from "we will test it next month" to "tests ran during this commit."

3.7.6 Deployment

In deployment: have deployment tools, a deployment script that supports continuous deployment, a release process that supports the deployment continuously.

The change: deployment stops being a manual ceremony and becomes a scripted, repeatable act. The same script deploys to every environment, the release process runs continuously, and the deployment can be rolled back as easily as it was performed. This is the phase where the anti-patterns of section 3.4 are finally removed: no hand-typed commands, no one-man dependency, no "did we remember step 23?"

3.7.7 Execution and Continuous Monitoring

In execution: achieve continuous monitoring throughout the project. Monitoring tools and technology proactively find the problem and actually respond to the error condition.

The change: the service in production is continuously observed — metrics, logs, and alerts are part of the system's normal operation, not an afterthought added when things break. Whenever it can happen, the response to an error condition is done in an automated, committed way: the monitoring detects the anomaly, the pipeline responds. If manual intervention is needed, it should allow interrupting the flow of continuous monitoring — a human can step in, diagnose, and take over, but the monitoring itself never sleeps.

3.8 Why DevOps Is Needed: The Bug Scenario and Beyond

3.8.1 How a Bug Report Travels: The Ticket Loop

When you implement software — when you convert an idea into a product — you have majorly three departments: the development team who develops it, the testers who test it, and the operation team who operates it. Consider system admins and every operation personnel as operators.

Now suppose a customer reports a bug. Who fixes that bug? Whenever a bug is there, a ticket is raised — and the ticket resides with the operations team, the maintenance people who support the product. The operations team does a few defined pre-checks: whether this service is on or not, whether the particular DB is accessible, whether the IP address of the data schema is accessible.

Forget about who will fix the bug — the main question for a client or a customer should be: when is this going to be fixed? Right now, in your team, the ball is getting shifted to another's court: "the problem is not my environment, the problem is your code" versus "the problem is not in the code, the problem is in the environment or in the configuration settings performed by you." This is one scenario where you do not know the timelines of fixing the bug if you do not have one team goal. The team is not one thing — they are segregated into their own zones.

3.8.2 The Working Imbalance

The next problem is an imbalance — a working imbalance, not a work-life balance. With agile, we work iterative and incremental. Say you released feature A, and now your team is working on feature B. Bugs are reported in feature A — bug one and bug two. The team has to meet the sprint goal: releasing feature B in two weeks. At the same time, the two bugs must be addressed, because per operational or business commitments, the bug raised by your customer has to be fixed. So what should the development team focus on — the new feature or the bug fix?

It is an imbalance. If they focus more on the new feature, the bugs will not be fixed. If they focus more on the bugs, the two-week sprint timeline will not be met — they have to reduce the scope. They are not robots; they are people — a resource, but not a hardware resource or an asset. You cannot expect everything from everyone at the same time.

Because of this imbalance you can see the lesser quality product. If they try to keep up both, they hamper the quality. To come out of it, they skip the good standard practices of code — they just fix it for the time being — and that impacts your code quality.

3.8.3 The Blame Game

And not the least: the blame game. Everybody blames each other. If there is a problem in the whole, the operation people are happy that the hole is not on their side. But you have to look at the whole picture: if something is wrong in your code, it impacts the project; and if it impacts the project, from the high business perspective it impacts the overall team. If the customer does not give you the budget for the next release or the next requirements, it impacts your whole business unit — and you all are in trouble.

The sailboat makes the point vivid: when a boat sinks, nobody is saved by being "not the one who made the hole." The whole crew shares the water. The same is true of a business unit: a production outage, a failed release, a lost customer — none of these respect team boundaries, and all of them eventually hit the budget of every team on the project. Blame does not fix the leak; it just spends the time that should be spent finding where the problem is. This is also why DevOps replaces blame with telemetry and transparency: when problems are visible to everyone, "whose side is it on?" stops being the question, and "what happened, and how do we prevent it?" takes its place.

3.8.4 The Complete Picture: Agile and DevOps Hand in Hand

Here is the complete picture. Until now: to break the wall between your business and your development team, we came with agile. Now, to break the wall between your development team and your operation team, we want to adhere to the DevOps culture.

In the diagram, DevOps and agile run hand in hand. Even in DevOps you need a process that should be followed. DevOps takes that process from agile, by adding a few extra best practices or manipulating, modifying some practices — but still it has the same agile base.

Picture the diagram: a horizontal line with business on the left and the customer on the right. Agile broke the first wall — between business and development — so business ideas became iterative work. DevOps breaks the second wall — between development and operations — so that iterative work flows all the way into production. The two practices overlap and run side by side, sharing the same iterative rhythm.

To break the wall between development and the operational people, convert this entire team into a DevOps team — not development, operation, tester, and so on. All members who work on a particular project are the DevOps team. That is where the need for DevOps comes from.

Exam Guidance Summary

Assignment for This Part

The assignment for this part: create two environments, one staging and one production, using git, and observe that once the code is changed it resides automatically through staging and production. You get the explanation in the course and the advantage of performing it by your own. Expect to build this yourself — the hands-on work is part of the evaluation.

Study Focus Points

These study points carry real weight in this material:

• The DevOps definition: a set of practices that reduce the time between committing a change and placing it into normal production, while keeping quality intact — including the people aspect and the wider scope beyond testing and development. Remember the three parts: practices, tools, and people.
• The meaning of done: done means the change is released into a production environment. Not "code is written," not "tests pass."
• The principles of software delivery: repeatable reliable process, automation, version control for everything, bring the pain forward, build quality in, shared responsibility (accountability differs from responsibility), continuous improvement.
• The problems of software delivery and the anti-patterns: manual deployment, deploying only after development is complete, testing on development machines, manual configuration management. The water tank lesson: correct practices applied at the wrong point produce nothing — find the real bottleneck.
• Focus on requirement gathering and understand the value stream map well before starting — a methodology that does not fit the problem statement will not give fruitful results.

Exam Notes Distributed Through This Lecture

• A methodology that does not fit the problem statement will not give fruitful results; understand the value stream map and focus on requirement gathering first.
• The definition of done is central: released to production.
• Accountability is different from responsibility; everyone shares responsibility for the delivery process.
• If it hurts, do it more frequently — bring the pain forward by picking the riskiest part first.
• Expect to build the staging and production environments with git yourself.

Key Industry Applications

Real-world: CI/CD pipelines appear in two variations in organizations — automatic flow from the git repository all the way to production, or automatic flow to staging with a manual push from staging to production. Facebook pushes code to production multiple times a day; Etsy performs dozens of routine deployments per day; IMVU deploys around 50 times a day with automated canary checks and rollback. Each of these is variation 1 taken to its logical end.

Real-world: version control (for example, git) is the safety net for everything — any misplaced data can be recovered from the previous version. In the enterprise, the same principle is applied to environments: configuration files, patch levels, deployment scripts, and even cloud templates are stored in version control so a whole environment can be re-created from the repository.

Real-world: infrastructure as code — deployment scripts and configuration as code — replaces manual configuration management in modern delivery setups. Tools such as Puppet, Chef, Ansible, and container platforms build environments from code; servers are rebuilt rather than repaired when they drift, and a misconfiguration in a script is fixed like a bug in application code.

Real-world: continuous monitoring tools proactively find problems and respond to error conditions, with manual intervention still allowed to interrupt the automated flow. Sites like Etsy and LinkedIn track tens of thousands to hundreds of thousands of production metrics, overlaying deployment events on their graphs so an anomaly can be traced to the change that caused it — the telemetry that replaces the blame game.

Real-world: methodology selection matters in practice — Agile, Scrum, XP, TDD, BDD, and FDD each fit different problem statements. A team building a compliance-heavy system with stable requirements may choose differently from a startup shipping a consumer app with uncertain requirements; picking a methodology that does not fit the problem yields unfruitful results.

Real-world: even today, some organizations still deploy only after development completes — exactly the anti-pattern described here. When you join an organization, the first thing to look at is its release pipeline: the presence of manual deployment, single-person configuration knowledge, or first-deployment-at-staging tells you which of this lecture's problems the organization still lives with.