Continuous Monitoring and the ELK Stack

16.1 What Is Monitoring?

16.1.1 Monitoring as a Process

The answer is monitoring (the process of observing and recording system state changes and data flow). The professor is deliberate about the word "process": monitoring is not a tool and not a system. A monitoring system — Nagios, Sensu, Amazon CloudWatch, the ELK stack — is the machinery you build to run this process, but the process itself is the idea you carry into any tool. Buy the best monitoring product in the world and you still have to decide what to observe, when to record, and what the recorded numbers mean. That deciding-and-observing loop is the process; the tool only executes it.

To observe and record, the process must capture two things:

1. State changes happening to your system or application — how the system moves from one condition to another.
2. Data flow in and out of your application, third-party applications, and your services — how data moves from one service to another.

Both are observed and recorded, and "recorded" means stored somewhere. An observation you do not store is gone the next second; you cannot compare today's behavior with last week's, and you cannot drill into an incident after it happened.

16.1.2 State Changes

A state change (the move of the system from one measurable condition to another) can be expressed in two ways:

1. Direct measurement of the state — read the value now, compare with the value before.
2. Logs recording updates that impact part of the state — an event message that says "this component changed to this value."

In plain terms, a state is some measurable description of your system at a moment in time, and a state change is when that description moves from one value to another. The professor's two examples make it concrete.

16.1.3 Data Flow

Data flow (the movement of requests and responses between components and systems) is captured by logging request and response data. When end users send requests to your application and your application responds, that exchange is data flow. It happens between internal components and between your application and external systems — and both kinds must be captured and stored.

• Internal data flow: in an e-grocery application, the search engine is your first service, and adding items to the cart is your second service. When a shopper searches for "wheat flour" and the search service hands the chosen item to the cart service, the data flowing from the search service to the cart service is internal data flow — it never leaves your application boundary.
• External data flow: your application sends a request to a third-party tool or library (say, a payment gateway or a mapping API) and receives a response. That request-response exchange with an external system is also data that needs to be captured and stored.

Why record both? A slow checkout can have two very different causes: the internal cart service is degrading, or the external payment provider is slow. Without logged request-response records on both sides of the boundary, you cannot tell which one is guilty. That distinction is exactly what root-cause analysis depends on.

Visual intuition: Picture a horizontal timeline of your application's condition. The x-axis is time (hours of the day); the y-axis is a health value (for example, CPU utilization as a percentage, 0 to 100). The healthy band sits near the 80 percent line. State changes appear as step changes in this line — a vertical jump from 80 to 90, a drop to zero when Tomcat dies. Data flow adds a second track beneath it: request counts per minute, with flat stretches during quiet hours and peaks at lunchtime. The monitoring process is what keeps both tracks continuously drawn. The takeaway: monitoring is a two-track recorder — one track for condition (state), one for movement (flow) — and an incident is usually visible as a disturbance in one or both tracks before any user complains.

16.1.4 Proactive Maintenance: Why Continuous Monitoring

In the traditional way of working, an application was maintained only after a ticket was raised by the end customer. Until the ticket arrived, nobody bothered about how the application was working. Then the team dug into the logs, found where the problem was triggered, and performed root cause analysis — after the customer had already suffered.

DevOps suggests the opposite: the maintenance part should be proactive, not reactive. The pipeline already has continuous integration, continuous code inspection, continuous testing, and continuous deployment and delivery — so why not continuous monitoring? Instead of waiting for a customer to raise something, the organization puts a rigid solution in place that automatically looks for problems and automatically resolves them. That is the mindset shift: catch the problem before the user reports it, and ideally fix it before it affects anyone.

The table summarizes the mindset shift: in the reactive world the ticket is the alarm; in the proactive world the monitoring system is the alarm, and the fix is often automatic by the time anyone looks.

Real-world & domain connection: in the banking domain, Kibana is most widely used as the visualization tool — we will see why later when we discuss the ELK stack, which is part of the course syllabus, free and open source.

Real-world & domain connection: This framing is not academic — it is the backbone of modern observability practice. In the banking domain, Kibana (the visualization half of the ELK stack, which is free and open source and part of this course's syllabus) is most widely used as the visualization tool; banks use it to render exactly this kind of recorded state and flow data into dashboards that regulators, trading desks, and operations teams can read. Every DevOps monitoring tool you will meet — Nagios, Sensu, CloudWatch, ELK — is simply a different engine for running the same process of observing and recording state changes and data flow.

16.2 The Goals of Monitoring and What to Monitor

16.2.1 The Five Goals of Monitoring

Monitoring has five distinct goals, and which goal you are pursuing decides what data you must collect closely. The five goals:

1. Identifying failures and associated faults — both at runtime (the failure is happening now) and during the postmortem (after a failure has occurred, reconstructing what happened).
2. Identifying performance problems — performance degradation of an individual system and of a collection of interactive systems: is the response time lagging? Are the services up to the mark?
3. Categorizing workload — workload categorization for managing capacity, both short-term and long-term capacity planning, and for the billing process.
4. Measuring end-user reaction to business offerings — how users like the features of the application.
5. Detecting intruders — monitoring and detecting anyone attempting to break the system, which matters most where security is a priority quality attribute.

16.2.2 Where the Data Comes From

The data you collect depends on the goal. The professor's source table is the exam-relevant summary of this section:

When we say infrastructure, we mean the hardware-level resources: more CPU, more computational power, more hardware resources. The application side means the software stack: your services, their threads, their queues, their configuration. When we talk about software capacity planning, we might need to increase thread pools or increase the runtime allocation of a service — that is application-level capacity. So capacity planning depends on both sources — application and infrastructure.

User reaction to business offerings is the exception: its source of data is only the application. Why? Because the end user does not bother about which infrastructure the application runs on. They want a working instance of the application; whenever they want to access a service they should be able to. Where the data is stored, from where the data is routing, what hardware resources are utilized — an end user never bothers about any of it. Their reaction is shaped entirely by what the application shows them and how it behaves: page loads, search results, checkout steps, feature layout. So for this goal the source of data is only the application.

16.2.3 Fundamental Items to Monitor

When implementing the monitoring process, three fundamental items should be considered — they form the input-processing-output loop that every application exhibits:

1. Input — what input is given by the end user. In an e-grocery application, the user clicks on the search box and types "wheat flour." That typed search is the input.
2. Resources — once the input is given, the services of the application process it. The search engine service processes the business logic written for search: it hits particular keywords and publishes the matching inventory items on the search results page. Processing involves resources:

• Hardware resources: CPU, memory, disk, and network — even though they are virtualized, they are still hardware resources (a virtual CPU is still consuming a physical core somewhere).
• Software resources: queues, thread pools, and configuration specifications.

1. Output — once the input is processed, the outcomes are returned: all the wheat flour items in the inventory get published. Output includes items such as transactions and business-oriented activities — all the outcomes.

Visual intuition: Draw the loop as three boxes in a line — Input → Resources → Output — with arrows between them. Under each box, note what monitoring records: under Input, request counts and request sizes; under Resources, CPU, memory, disk, network, queue depth, thread-pool utilization; under Output, transaction counts and business outcomes. The five goals then attach to different parts of the loop: failure detection watches all three, performance watches Resources, user reaction watches Input and Output, workload categorization watches Input (how much work is arriving), and intrusion detection watches for abnormal Input and abnormal Output patterns. The takeaway: every monitoring goal is a way of reading this one input-resource-output loop.

Real-world & domain connection: In industry, this goal-to-source mapping is exactly how monitoring stacks are designed: infrastructure monitoring tools (CloudWatch, Nagios) feed hardware-level data, while application performance monitoring (APM) agents feed the application-level data, and product analytics tools (web analytics, session replay) feed the user-reaction view. The exception in the professor's table is why e-commerce companies run three separate tool families — a bank or an e-grocery platform cannot infer a customer's reaction from a CPU graph, and equally cannot diagnose a slow checkout from click analytics alone.

16.3 Goal 1: Failure Detection

16.3.1 Hardware Failures: Total vs Partial

Failures of any element in physical infrastructure are possible. The cause can be anything — overheating, wear, loose connections, even physical damage to cabling. What matters for monitoring is not the cause but the shape of the failure, and the professor's example is the cable pin in a socket.

Real-world & domain connection: Nobody wants to purchase their own server and configure it anymore; everybody looks for on-demand infrastructure — third-party services. When you acquire third-party services, hardware monitoring responsibility lies with the data center provider. For example, on Amazon's AWS cloud platform, hardware monitoring is done by the cloud provider itself — AWS watches the physical hosts, network, and storage fabric, while the customer monitors what runs on top. In practice this means the partial hardware failures of the provider's fabric often surface to the customer only as mysterious application-level slowdowns — exactly the "response time changes" symptom of the loose pin, now observed across an organizational boundary.

16.3.2 Software Failures: Dependency Failure and Misconfiguration

Hardware is only half the story. Software can also fail, either totally or partially — and there are two primary reasons for software failures:

1. Dependency software failure — the application depends on another software component, and that component fails. Example: an application service depends on Tomcat. If Tomcat is shut down, there is no response and the service will not work. The application itself never changed — its dependency died, and the application died with it.
2. Software misconfiguration — the application is set up incorrectly. Example: the application depends on, say, JRE 7 (Java Runtime Environment version 7), and due to a manual mistake or a script mistake the JRE 7 is upgraded to 8. The result is compatibility issues or software failures due to the misconfiguration. The runtime is present and running — it is just the wrong one.

16.3.3 Three Ways to Detect Software Failures

Detecting software failures can be done in one of three fashions:

1. Monitoring software (external health check) — the monitoring software performs a health check on the system from an external point. The monitor stands outside the application, sends probes in, and reads the response: is the port open? Does the endpoint answer correctly?
2. A special agent (in-system monitor) — an agent residing inside the system itself collects data and cross-verifies whether the system is in the desired state: whether dependent resources are up and active, and whether software configuration settings have changed. Because the agent lives inside, it can see what an external probe cannot — internal queues, thread pools, configuration files, dependency state.
3. Self-detection (the system reports on itself) — the system itself detects the problem and reports it. The application catches its own errors, logs them, and notifies the monitoring system.

These are the three different ways of detecting software failure. Note how they line up with the earlier material: the external health check and the in-system agent correspond to the health-check and agent-based monitoring we will meet again in the "How to Monitor" section, and the misconfiguration example above is precisely the kind of check only the in-system agent can do well.

Visual intuition: Picture the monitored system as a house with three kinds of observers. Observer 1 stands outside the gate and knocks periodically — if the door opens, the house is "healthy" (external health check). Observer 2 lives inside the house and walks every room, checking that the boiler, the wiring, and the configuration of every appliance match the blueprint (in-system agent). Observer 3 is the house itself — a smoke detector wired to an alarm (self-detection). Total failure (the house burned down) is obvious to all three; partial failure (the boiler runs but heats poorly) is visible only to observers 2 and 3 — which is why monitoring design chooses which observers to deploy per component.

Real-world & domain connection: This total/partial distinction shapes how real sites do incident detection. Health checks and heartbeat monitors catch total failures instantly, while the "loose pin" class of problems — flaky networks, degrading disks, dying memory — is what modern SRE practice hunts for with golden-signal monitoring (latency, traffic, errors, saturation). In the cloud era the physical half of this responsibility moved to the provider: AWS, Azure, and Google Cloud monitor their own hardware fabric (the cloud provider's responsibility), while tenants monitor their applications and the configuration drift that so often breaks them.

16.4 Goal 2: Performance Degradation

16.4.1 Detecting Degradation: Current vs Historical

Degraded performance can be observed by comparing the current performance with the historical data. Is the response intact like it was earlier, or is there a delayed response? The comparison needs history — which is why the "recorded" half of the monitoring process from Section 16.1 matters so much: without stored historical data there is nothing to compare the present against.

Scope — what the current-versus-historical comparison assumes: The comparison assumes the past is a valid baseline for the present. If the workload itself changed (a new feature shipped, a marketing campaign doubled traffic, the user base grew), the historical numbers are no longer a fair reference — a "degradation" may simply be a heavier workload. This is why the worked example in this section must be read carefully: going from 50 to 45 completions of the same operation per hour is only a degradation if the demand is unchanged. Professionals couple the per-hour completion count with user counts before declaring a problem.

16.4.2 Performance Metrics: Latency, Throughput, Utilization

Several metrics help measure performance. Three form the standard performance picture:

Latency is the time from the initiation of an activity to its completion — the period from user request to satisfying that request. In the running example: you search for wheat flour; how much time did the system take to give you the results? That is the latency. It can be measured at various levels of granularity: at a coarse grain it is the period from a user request to the satisfaction of that request; at a fine grain it is the period from placing a message on a network to the receipt of that message.

$$L=t_{\text{end}}-t_{\text{start}}$$

where:

• $L$ is the latency — the time the activity took, measured in time units (seconds, milliseconds).
• $t_{\text{start}}$ is the time the activity started — the clock reading at initiation.
• $t_{\text{end}}$ is the time the activity completed — the clock reading at completion.

The formula is a simple difference of two clock readings. It works cleanly when both readings come from the same machine; across different machines, clock synchronization makes it harder (this is why distributed tracing uses coordinated timestamps). Latency is cumulative: the latency of satisfying a user request is the sum of the latencies of all sub-activities until the request is satisfied, adjusted for parallelism — so knowing the latency of each sub-activity tells you where a slow request lost its time.

Throughput is the number of operations of a particular type in a unit of time. Example: report generation is one particular type of operation; in one unit of time, how many report generations happen? Previously the application did 50 report generations in the unit time period; currently only 45 happen. That drop is a performance degradation.

$$T=\frac{N_{\text{ops}}}{\Delta t}$$

where:

• $T$ is the throughput — operations of the particular type per unit of time.
• $N_{\text{ops}}$ is the number of operations of the particular type that completed.
• $\Delta t$ is the duration of the observation window — the unit of time.

Throughput has a single-user cousin, latency: throughput is a system-wide measure involving all users, while latency is per request. High throughput does not imply low latency — a system can serve many requests slowly, or few requests quickly — and the relation depends on the number of users and their pattern of use.

Utilization is the relative amount of use of a resource. For hardware resources: CPU utilization, memory, disk. A useful threshold example: CPU should utilize 80 percent; if it increases beyond that, to 90 or 100 percent, you definitely see performance degradation in the application, because the computational power is being used hugely on one particular service and you are not getting the response.

$$U=\frac{r_{\text{used}}}{r_{\text{total}}}\times 100\%$$

where:

• $U$ is utilization as a percentage.
• $r_{\text{used}}$ is the amount of the resource currently in use.
• $r_{\text{total}}$ is the total amount of the resource available.

A quick sanity check of the formula: if 80 cores are in use out of 100 available cores, $U=\frac{80}{100}\times 100\%=80\%$ — a ratio that always lands in $[0,100]$, as a percentage must. Utilization makes sense only when usage is attributed: "the application is using 80 percent of CPU" is actionable, while "CPU is 80 percent used" without knowing what is consuming it supports no decision.

Visual intuition: Plot latency, throughput, and utilization on one dashboard against time. Latency is the line that stays flat during health and rises (with spikes) during trouble. Throughput is the line that follows demand — it breathes up and down with the business day. Utilization is the line that hugs the ceiling when a resource is saturated. The tell-tale pattern of degradation: utilization climbs toward 100 percent while latency starts rising and throughput stops climbing — the road is full, the travel time is up, and the toll-gate count has flattened. The one-sentence takeaway: no single metric tells the story; the trio together separates "heavy but healthy" from "saturated and failing."

Utilization is the third metric; latency, throughput, and utilization together give you a performance picture. Memory and disk are hardware resources measured this way; the number of queues or thread pools created are the software resources measured this way.

16.4.3 Worked Example: Report Generation Throughput Drop

16.4.4 Worked Example: CPU Utilization Threshold

Visual intuition: Imagine a line chart of CPU utilization with time on the x-axis and percentage on the y-axis (0-100 percent). During normal hours the line rests near 80 percent with small jitter. When the runaway service starts, the line rises through 90 percent and pins against the 100 percent ceiling, flattening at the top — a saturated line is the classic saturation signature. The latency line above it rises in the same window. The takeaway: utilization crossing the threshold together with rising latency and flat throughput is the confirmed degradation pattern.

Real-world & domain connection: These three metrics are exactly the golden signals that site reliability engineering teams monitor in production — latency, traffic (throughput), errors, and saturation (utilization). The Platformer.com case in the reference text shows them in action: a customer's servers ran at a normal CPU utilization of about 5 percent, then spiked to around 17 percent over two days — a pattern suspicious enough to look like an intruder, but which turned out to be the customer's opening-night event. That case illustrates both the power of threshold-free, baseline-relative reading of utilization and the cost of missing application-level context. In banking, the same trio powers service-level agreements: a payment gateway contract specifies latency (p99 response under 500 ms), throughput (thousands of transactions per second), and utilization ceilings for capacity review.

16.5 Goal 3: Capacity Planning

16.5.1 Long-Term Capacity Planning

There are two types of capacity planning: long-term and short-term.

Long-term capacity planning needs human involvement — even though the monitoring is continuous, a human is needed here — and works on a time frame of days, weeks, months, or even years. It is intended to match the hardware needs, whether real or virtualized, with the workload requirement: what workload are you observing on your application, and based on that, how much infrastructure should the organization increase?

In a physical data center, long-term planning involves ordering new hardware. In a virtual public data center — which is what almost everybody uses now — it involves deciding how many new virtual machines to request, based on how many users are registered and accessing the virtual resources. The categorization of the current workload comes from the monitoring data: how many users are getting registered and accessing your application. From that current workload plus the business consideration, you project the future workload, and the organization decides whether to increase capacity.

16.5.2 Worked Example: The Online Paper Distribution Startup

16.5.3 Short-Term Capacity Planning

Short-term capacity planning is done automatically — there is no human interaction — and the time frame is in minutes, hours, or within that day. It depends completely on the cloud platform context: third-party vendors, on-demand infrastructure.

In short-term capacity planning, capacity decisions happen like creating new virtual machines or shutting down existing virtual machines, in terms of meeting the billing purpose — optimizing the cost. Monitoring the usage of current VM instances is an important part of this. The idea exists because charging for use (pay per use) is an essential characteristic of any cloud provider, a term defined by NIST — the U.S. national body for science and technology standards, known by the acronym NIST. The question is: how much optimized a way can we use the cost and infrastructure? Because every running VM is metered and billed, a VM that serves no users is not a safety cushion — it is a cost line.

16.5.4 Worked Example: Amazon's VM Shutdown at Night

Visual intuition: Plot the number of active VMs and the number of concurrent users on the same chart, with time of day on the x-axis. The user curve has a mountain during the day and a valley at 1–5 a.m. The VM line follows it: 100 during the day, stepping down to 50 in the valley, stepping back up at dawn. On the weekend, the whole profile lifts. The takeaway: capacity follows demand like a shadow — automatically in short-term planning, after a human decision in long-term planning.

Real-world & domain connection: This is how every major cloud consumer runs cost and capacity. Amazon's own e-commerce platform and thousands of organizations use autoscaling rules driven by monitoring to shrink fleets at night and grow them for weekend peaks; NIST's pay-per-use definition is literally the legal and economic foundation of cloud billing (metering usage is how AWS, Azure, and Google Cloud can charge by the hour and by the GB). Capacity projections built from monitored workload data also feed budget cycles — the "order more infrastructure" decision in the XYZ example is a quarterly business process in real companies.

16.6 Goal 4: User Interaction Monitoring

16.6.1 What to Observe

The fourth goal is to observe how users interact with the application. Three things need observation:

1. Latency of user requests — how long each user request takes. Users expect decent response times, and the impact of delay is measurable: even a few hundred extra milliseconds changes how much users use a service.
2. Reliability of the system — if you increase the number of users accessing a particular service, does the system respond correctly or does it get trashed (degraded)? Reliability is watched under load: the question is not "does it work?" but "does it keep working as the crowd grows?"
3. User interface modification — when the user clicks on the search engine and types wheat flour, does the next pane update with the published wheat flour items? The UI must actually change as the user interacts — the pane updating with the search results is the visible proof that the request-response loop reached the user's screen.

All of this is the user monitoring goal. Note the echo of the earlier metrics: the first item is the latency formula of Section 16.4 applied at the user level — $L=t_{\text{end}}-t_{\text{start}}$, with the start being the user's click and the end being the user's screen update.

16.6.2 Real User Monitoring vs Synthetic Monitoring

There are two ways of doing user interaction monitoring:

1. Real user monitoring — real end users are accessing the application, and you monitor the data flow from the end user. Every actual user interaction is captured: every click, every request, every response time. Because the data comes from genuine users, it reflects the real service level users experience — including their real network conditions, browsers, and locations. Real user monitoring is usually passive: it watches without injecting load or changing the server-side application.
2. Synthetic monitoring — generally performed by the organization for capacity testing and user acceptance testing. A script runs and creates multiple instances of the same service in your application, creating artificial users with the help of the script; the script hits the particular services of your application to see whether the application is responding in time and whether performance degrades. Because the script drives the same scenario every time, results are systematic and repeatable — it does not matter what real users happen to be doing right now.

Real users on one side, artificial script-driven users on the other. The two approaches differ on exactly the dimension that matters:

When to pick which: use synthetic monitoring to verify behavior systematically (before releases, during capacity tests), and real user monitoring to learn what users actually experience in production — most organizations run both, because each covers the other's blind spots.

Visual intuition: Picture two monitors side by side. Left screen: a stream of dots, each dot one real user session, appearing at random moments with different speeds and paths — noisy, real, alive (real user monitoring). Right screen: a metronome — identical probes firing every five minutes on a fixed route through the application, each exactly like the last (synthetic monitoring). The metronome finds regressions the moment they happen (it always runs), while the stream shows the truth about real conditions (it always reflects reality). The takeaway: one gives you a heartbeat you control; the other gives you the pulse you did not ask for.

Real-world & domain connection: This goal is a core practice of e-commerce and banking sites: real user monitoring feeds the dashboards that show p95 and p99 page-load times, and synthetic monitors ("synthetic checkers") probe critical journeys — login, search, checkout — around the clock from multiple locations. The stakes are documented: delaying a search results page by even 100–400 ms measurably reduces how many searches users perform. In banking, user interaction monitoring on the customer portal pairs with the intrusion-detection goal of the next section — the same user sessions that tell you the experience is smooth can also reveal sessions that do not match the user's normal behavior.

16.7 Goal 5: Intrusion Detection

16.7.1 Role-Based Activity Monitoring

Intruders can break into a system either by disturbing an application by providing incorrect authorization. With this goal, the application monitors users and their activities to determine whether those activities are in line with the user's role in the organization.

Role-based monitoring works by pairing two things:

1. The role — the set of activities the organization has assigned to a user (admin, operator, customer support, reader).
2. The observed activity — what that user's account actually does.

When the observed activity departs from the role's expected pattern, the deviation itself is the alarm trigger — no signature of a known attack is needed.

16.7.2 Network Traffic Monitoring

An intrusion detector is a software application that monitors network traffic by looking for abnormalities. The abnormal can be caused by attempts to compromise the system, or by violations of the organization's security policies — it does not need to wait for a successful break-in.

16.7.3 Technologies Intrusion Detectors Use

Intrusion detectors use a variety of technologies to identify attacks:

1. Historical data from the organization's network, to understand what is normal. The application must first understand what exactly the normal is — the baseline (Pune location, 5 p.m. report, usual traffic volumes) comes from recorded history.
2. Libraries containing network traffic patterns observed during various attacks, so the system can understand what an attack can look like — the known-attack signatures that a pattern-matching engine can recognize.

These two knowledge sources are compared against current traffic to make the detection decision.

Visual intuition: Picture a time series of network traffic volume — the x-axis is time of day, the y-axis is packets or requests per minute. The normal band is a gentle curve that breathes with the business day. An intrusion in progress appears as a spike breaking out of the band, or a plateau where there should be a valley, or steady traffic from a location that never contributes to the curve. The detector draws the normal band from historical data, checks the current point against it, and rings the alarm when the point escapes the band — while the attack library cross-checks the shape of the escape against known attack silhouettes. The takeaway: detection is a comparison — current against historical normal, current against known attack patterns.

Real-world & domain connection: This goal is the essence of modern security operations: user and entity behavior analytics flag accounts that deviate from their own baselines, and network intrusion detection systems (IDS) compare live traffic against attack-signature libraries. Banks and payment providers run exactly the Pune-Amsterdam play: logins from unexpected geographies trigger step-up verification or customer messaging rather than hard locks, and authentication platforms add device fingerprints and behavioral scoring. The role-based half maps to privilege monitoring — an admin account suddenly running bulk export jobs at 3 p.m. is a classic exfiltration tell in insider-threat programs.

16.8 How to Monitor an Application

16.8.1 Health Checks and Heartbeats

A health check uses a heartbeat signal: you send a heartbeat signal to a particular service or resource of your application and see the response back — whether it is a bit 1 or bit 0. If it is 1, that means healthy; if 0, unhealthy — or vice versa, depending on your convention.

$$h\in \{0,1\},h=1\Rightarrow \text{healthy},h=0\Rightarrow \text{unhealthy}$$

where:

• $h$ is the health indicator — a single binary bit returned by the heartbeat probe.
• $h=1$ means the service is healthy (the response came back as expected).
• $h=0$ means the service is unhealthy (the response was missing or wrong).
• The convention is a matter of choice — some systems use 0 for healthy and 1 for unhealthy — but the meaning is fixed by definition, not by the wire format, so consistency matters more than which way round the bits go.

The health check is deliberately coarse: one bit answers "is it alive and answering?" — it does not tell you how healthy, which is what the performance metrics of Section 16.4 are for. A spot check of the formula's behavior: send a probe to the Tomcat instance from Section 16.1 — port open and responding means $h=1$ (healthy); no response within the timeout means $h=0$ (unhealthy).

Health checks happen within a timeline: you send the heartbeat signal and in that particular time frame you get the result back. If even the time frame gets delayed, you can see there is a degradation in performance — the heartbeat itself becomes a latency probe. A service that answers every time but answers late is telling you something: it is alive, but it is struggling.

16.8.2 Agent-Less vs Agent-Based

Before choosing a mechanism, recall what monitoring must do as a whole: monitoring is collect the data, store the data in a particular format, and when you want to visualize, the monitoring system should help you visualize the data. Collection, storage, visualization — the agent question is about the collection step.

• Agent-less means you do not need a particular agent residing inside the application; the monitoring system performs the monitoring without installing any agent inside your software application. The system is observed from outside — through protocols it already exposes (a metrics endpoint, an operating system service, a management protocol). Nothing new is installed; you use what is already there.
• Agent-based means you need to install an agent into your software system. The system includes the application, the middleware, the OS, and all the hardware resources; the agent is installed on top of all the components of your system. The agent lives inside, reads internal state directly, and ships it to the monitoring system — it sees what external probes cannot.

16.8.3 Monitoring and Configuration Management Work Hand in Hand

Monitoring and configuration management are two halves of one loop. If there is a state change, you want the state to redirect back to the desired state — that is what automation wants to achieve. To do that you need a configuration management tool, and that is where the monitoring tool and the configuration management tool work hand in hand: the monitoring tool detects the departure from the desired state, and the configuration management tool restores it.

A monitoring system provides:

• Visualization — store monitoring data on different platforms like big data or traditional BI, and visualize it nicely.
• Intrusion detection support — the stored historical data can be compared with the current data.
• Business decisions — monitoring data is sometimes used to take business decisions and launch new features that can hit the market.
• User tracking — capture operation logs and store them.
• Configuration management support — the configuration management system helps push the system back to the desired state if there is any state change.
• Alarm evaluation — set threshold values: if CPU utilization hits 80 percent, send alerts; if it increases to 85 percent, send alarms to the operators. Automated alarms and alerts are triggered to the operators and other systems, and this happens with automation supported by the monitoring system.

Broadly, the three main features of a monitoring system are: visualization, triggering alarms and alerts, and storing the monitoring data in a particular format — everything else on the list is a use of these three.

Visual intuition: Draw the detect-restore loop as a circle: on the left, the monitoring system probes the application (health check bit $h$, metrics, logs); on the right, the configuration management tool pushes the application back to the desired state; at the bottom, the stored monitoring data feeds visualization dashboards and alarm evaluation; at the top, the alarms reach the operators and other systems. The arrows run one way from the application into monitoring (collect), one way from monitoring to visualization and alarms (present), and one way from configuration management back into the application (restore). The loop only closes when all three arrows exist — the professor's point that the two tools "work hand in hand."

Real-world & domain connection: These mechanisms are exactly what runs in production today: load balancers and orchestration platforms (Kubernetes) fire liveness and readiness probes — the heartbeat bit of this section — at every container, while agents (like CloudWatch agents or APM agents) are installed on servers for deep metrics, and agent-less collection uses protocols such as SNMP for network gear. The monitoring-configuration-management loop is the standard automation pairing: the monitoring tool detects drift or failure, and the configuration management tool (Chef, Puppet, Ansible — met in Section 16.10) converges the system back to the desired state, giving banks and e-commerce platforms self-healing fleets instead of overnight pager duty.

16.9 Why Continuous Monitoring

16.9.1 Error Diagnosis and Root Cause Analysis

A monitoring system allows operators to drill down into detailed monitoring data and logs. The first benefit is error diagnosis: you can easily diagnose the error and perform root cause analysis using all this data. "Drill down" is the key phrase — the operator starts at the top-level symptom (the alarm on the dashboard) and descends through layers of data: which service failed first, which request was in flight, what the logs say at that exact moment, which resource crossed its threshold. Because the data was stored continuously (recall the record half of the monitoring process), the timeline of the incident is complete — nothing has to be reconstructed from memory, and the postmortem can replay what happened rather than guess at it.

16.9.2 Deciding the Best Reaction

The second benefit is deciding the best reaction to the problem. Do we need to ignore the problem? Do we need to solve the problem? Do we need to give a hot fix, or do we need more time to fix the problem with proper process? All of this is supported by the drill-down into monitoring data.

The reaction decision depends on severity — and severity depends on data:

• Ignore: the anomaly is below the alarm threshold, or it is a false positive (a legitimate event that looked unusual). Only the data can tell you this is safe.
• Solve now: the incident is active and customer-impacting — a hot fix or immediate remediation is justified.
• Fix with proper process: the problem is contained or non-urgent — a root cause analysis, a tested fix, and a normal release cycle.

Visual intuition: Picture the drill-down as a funnel. At the top, one alarm: "CPU utilization above 85 percent." One level down, the service list: the report generation service is the consumer. Next level, the request traces: a batch of requests waiting on a database query. Bottom level, the logs: a slow disk on one node. Each level narrows the suspect set, and the funnel is only as deep as the data that was stored. The takeaway: monitoring data converts a single alarming number into a path from symptom to cause.

Real-world & domain connection: This is the post-incident discipline of every serious engineering organization: incident postmortems are built on stored monitoring data, dashboards and logs (the same ELK stack that appears later in this lecture), and the ignore/solve/process distinction is formalized in severity classification systems — Sev 1 (customer impact, fix now) through Sev 3 (fix in a normal release). The Platformer.com case from the reference material is a live example: what looked like an intruder spike turned out to be a customer's opening night — and only application-level data, added to the CPU-level data, could resolve the false alarm and choose the right reaction.

16.10 Monitoring Operation Activities

16.10.1 Monitoring Configurations and the Desired State

Operation tools monitor the resources, like your configuration settings, to determine whether they conform to the desired state or desired setting. They also monitor resource specification files to identify if there are any changes. Both types of monitoring are best done by agents that periodically sample the actual values and the files that specify those values — the periodic sampling is what turns "the configuration changed at some point" into "the configuration changed at 03:12, between these two known-good snapshots."

The configuration management tools: Chef, Puppet, and SaltStack are agent-based configuration management tools, whereas Ansible is the agent-less configuration management tool. These tools help find out whether there is a change in the desired state, and if there is, the configuration management tool brings the system back to the correct state — closing the detect-restore loop opened in Section 16.8, now with named tools.

16.10.2 Collection and Storing of Data

The core of monitoring is to record and analyze time series data (a sequence of timestamped data points — when the request was seen, what the process ID was for that activity). All of this has to be stored. There are three key challenges in collection and storing of data:

1. Collecting related items by time — your system may be a distributed system where the components are not consistent; identifying the related services by time becomes difficult. Clocks on different machines drift (microseconds within a cluster, far more across clusters), so deciding that two events are "at the same time" — or even which came first — is problematic.
2. Collecting related items by context — if there is any parallel process for the same operation, it becomes difficult to reconstruct the sequence of those events. Example: report generation is the operation that has a problem; your monitoring activities generated logs, but there are two parallel report generation activities and only one has the problem. It becomes challenging to reconstruct the sequence of events to diagnose what went wrong — without a request ID tying each log line to its specific report generation, the two parallel sequences are indistinguishable.
3. Handling the volume of monitoring data — you get a lot of data. Storage systems like big data and Hadoop can help handle the large volume of monitoring data, with retention policies to keep fine-grained recent data and coarser aggregates of older data.

The great solution to all three: change your monitoring configuration over time. Your application is changing very frequently, so you need to change your monitoring configurations as well, in line with the change happening to your code base — finer sampling during risky windows (upgrades, deployments), coarser sampling once things settle, and re-set thresholds when the system's normal changes.

16.10.3 Logs: Sources, Uses, and Best Practices

Logs are generally generated with the help of your software — wherever you write down some information to store it.

Sources of logs: your application, the web servers you use, database systems, all tools and technology used in your application, the DevOps pipeline itself, operational tools, upgrade tools, migration tools, and the configuration management tool. All of these generate logs — every actor in the ecosystem writes a record of what it did.

When do we use these logs?

• During operation, to detect and diagnose the problem — if an alarm, alert, or any issue got triggered, you have to diagnose from where it was triggered, detect the issue, and resolve it, using the stored logs.
• During debugging, to detect the errors.
• During the postmortem / problem forensic, to understand the sequence that led to the particular problem.

The professor's closing argument for the rules: if we have best practices for writing code and best practices for writing test scripts, why can't we have best practices for writing logs? Logs are code — they deserve the same discipline.

16.10.4 Graphing and Displaying Data

The next operation activity is to graph and display the data with nice visualization. It is useful to visualize all the relevant data collected by the monitoring system.

Real-world & domain connection: this is a great help in the banking domain to understand the stock market — how the stock is getting exchanged, the state change in the stock. Visualization also helps you understand how much capacity is getting used — the utilization of your application. It helps display this information broadly to higher authorities in terms of finalizing the upcoming business offerings. Since you have a great amount of data, why not visualize it in a graphical view and take appropriate decisions? The graph is the decision surface: patterns that are invisible in raw numbers (spikes, trends, saturation plateaus, cyclic variation) become readable at a glance, and an experienced operator reads the shape of a graph the way a doctor reads an ECG.

16.10.5 Alarms and Alerts

The monitoring system should auto-generate alarms and alerts.

Alerts are raised for the purpose of informing — it is just an inform. Alerts are raised in advance of an alarm, and every alarm will have an alert. Alarms require action by the operator or by another system.

The threshold pattern — the exam-relevant formula. With $u$ the CPU utilization as a percentage:

$$u\ge 80\%\Rightarrow \text{alert},u>85\%\Rightarrow \text{alarm}$$

• $u$ is the CPU utilization, measured as a percentage.
• At or above 80 percent: an alert is raised — informing the operators that the resource is climbing toward danger.
• Strictly above 85 percent: an alarm fires — the threshold for required action has been crossed.

The boundary convention is a configuration choice; what matters is the ladder: the alert threshold is set below the alarm threshold so that operators are informed first and ordered to act second. Once the alarm is there, the operator has to deep-dive and look into the problem to minimize the CPU utilization, to bring the value back to normal.

Visual intuition: Picture a two-tier threshold line on the CPU utilization graph: a lower dotted line at 80 percent and a higher solid line at 85 percent. As the utilization curve climbs, it first crosses the dotted line — the alert fires, an inform — and if it keeps climbing through the solid line, the alarm fires, and the operator's phone rings. On the Saturday curve, the same 80 percent crossing rings the phone directly because the context rule lowered the action line for the peak day. The takeaway: the graph shows values; the thresholds show intent.

Real-world & domain connection: These exact patterns run in every serious operation. Chef, Puppet, and SaltStack converge node configurations and monitor drift, while Ansible pushes the same policies agent-less; time series databases (Prometheus, InfluxDB) and big data stores (Hadoop) absorb monitoring volume; structured logging with request IDs is the industry standard that makes distributed debugging possible; and banking dashboards built on Kibana (the ELK visualization tool of Section 16.12) render stock movements, utilization, and business metrics for decision-makers. The ticket-to-alarm mapping is how ITIL-based ticketing systems and modern incident tools (PagerDuty-style alerting) reconcile with a developer's world: the ticket you fix is the alarm the monitoring system raised, and the alerts are the evidence trail for your root cause analysis.

16.11 Challenges of Monitoring with DevOps

16.11.1 Monitoring Under Continuous Change

The first challenge: monitoring under continuous change. DevOps is nothing but continuous change — continuous integration means even a small change to the code base triggers the whole pipeline and the code resides in the production environment. How can we monitor under continuous change?

The solution: automate the configurations of alarms, alerts, and thresholds as much as possible. The DevOps process is already automated — code is pushed continuously — so why not automate the monitoring process activities that used to happen manually, like configuring alarms and alerts? The monitoring configuration process is just another DevOps process.

16.11.2 Bottom-Up or Top-Down Approach

The second challenge: do we follow the bottom-up approach or the top-down approach in monitoring?

• Bottom-up: data is collected from all the levels — application component, operating system, physical component (hardware resources), and application resources — but the close observation happens at the physical component, the hardware level. The watchful eyes are at the bottom of the stack.
• Top-down: data is also collected from all levels, but the close observation of the monitoring system happens towards the application component. The watchful eyes are at the top, at the user-visible layer.

When to pick which: With DevOps, everybody is moving to a cloud platform. If it is SaaS or PaaS, you are not aware about the infrastructure and hardware very well. So does it make sense to follow the bottom-up approach? No — because you are not really bothered about the hardware resources; you need to think of a top-down approach. But if you have IaaS — infrastructure as a service — you are dealing with the configurations of the infrastructure as well, so you can think of both bottom-up and top-down.

16.11.3 Monitoring a Microservice Architecture

The third challenge: monitoring into a microservice architecture. Microservice means your application itself is decoupled — you have multiple services in your application that are loosely coupled. A request is processed by each and every service of your application. If there is a problem in one particular service, it becomes challenging to identify from where the problem has been triggered in a microservice architecture: the same slow response can be produced by a slow service, a slow dependency, a slow network hop, or a saturated queue anywhere along the chain.

The solution: you can have an intelligent monitoring system. Whatever tools and technology we have in the market, they all support monitoring at the microservice level as well. The modern tooling traces a request through every service it touches, so the operator can follow the path the request actually took — which is the only reliable way to say which of the dozen services was the one that failed.

16.11.4 Dealing with Large Volumes of Data

The fourth challenge: dealing with the large amount of data. We already discussed Hadoop and big data systems for storage. But every time, the full data is not useful — there could be data which is just dumb, just collected. Can we minimize the collection of data?

Yes. The cadence solution — make the collection interval fit the situation:

Visual intuition: Picture the sampling grid as a timeline with tick marks. During the upgrade window, the ticks crowd together (10-minute spacing) — dense data, every blip visible. After 15 days, the ticks spread out (30-minute spacing) — sparse data, trends still visible, blips smoothed over. Two graphs of the same metric: the first jagged and detailed, the second smooth and calm. The takeaway: collection cadence is a knob you turn — dense when the risk is high, sparse when the risk is low.

Real-world & domain connection: These four challenges are the research agenda of modern monitoring vendors and the daily reality of platform teams. Autoscaling and infrastructure-as-code make automatic server registration standard practice; SaaS/PaaS customers live top-down whether they choose it or not, while IaaS operators combine both; microservice tracing (distributed tracing with request IDs, as championed by tools like Jaeger and Zipkin, and baked into Kubernetes observability) is the accepted answer to the third challenge; and variable collection cadence is how large fleets control their telemetry bill — the exact economics the professor motivates with the 10-minute-to-30-minute example. ELK, the next section's topic, is the storage-and-visualization backbone that most organizations pair with these practices.

16.12 Monitoring Tools and the ELK Architecture

16.12.1 The Tooling Landscape

There are a lot of tools and technologies in the market; the diagram of tools and technology in the DevOps environment includes Sensu, Nagios, Amazon CloudWatch (referred to as "Cloud Wars" in the discussion), Logstash, Kibana, and more. Each occupies a different niche: Nagios is a classic alerting-focused monitor with a large plug-in pool; Sensu is its more scalable, cloud-friendly successor; Amazon CloudWatch is the monitoring service that AWS provides inside its own cloud, collecting hundreds of metrics at a fixed interval; and Logstash and Kibana belong to the ELK family that the lecture now focuses on.

The last topic of the syllabus: ELK. ELK is the acronym for three open source tools: Elasticsearch, Logstash, and Kibana. These three different tools are clubbed together into one solid solution, and all are free and open source — which is why they are part of the course handout. The ELK architecture is very commonly used by multiple organizations, and in the banking domain Kibana is most widely used as the visualization tool.

16.12.2 Elasticsearch

Elasticsearch is a search engine: open source, distributed, and it works with a RESTful API. It is a JSON-based key-value pair search engine. With this search engine you can search any type of data — structured, unstructured, geometric — any kind of data can be searched with Elasticsearch. Under the hood it is a NoSQL database that is based on the Lucene search engine — Lucene is the Java search library that provides the inverted-index machinery, and Elasticsearch wraps it in a distributed, RESTful, JSON-friendly layer.

16.12.3 Logstash

Logstash is again open source, a server-side data processing pipeline that collects the data, pre-processes it, transforms it to a particular format, and sends it to the stash — from where you can search the data. It can collect data from any device: routers, web servers, network routers, any device in your application.

16.12.4 Kibana

Kibana helps with the visualization of charts and graphs — it visualizes your Elasticsearch data and navigates Elasticsearch. Kibana gives you the freedom to select the way you give shape to your data. Kibana core ships with the classic histograms, line graphs, pie charts, sunburst, and many more ways of visualizing your data.

16.12.5 The ELK Architecture: How the Pieces Connect

The architecture: Logstash can collect data from any kind of device — wire data, log files, metrics, Windows events — and those are passed and transformed to Elasticsearch. You can even think of shipping data to Elasticsearch without Logstash; Elasticsearch can search all these kinds of data. But to give a proper shape to your data, transform it, pre-process it, and minimize the data size, people use Logstash to transform the data into a particular format and then store it into the stash.

Quick review of the flow: logs — whether from wired data, routers, databases, or any other files — are collected by Logstash, pre-processed, and stored in one particular format into Elasticsearch. By using Elasticsearch you can search for particular data, and that data can be visualized with Kibana. That is how ELK works.

Visual intuition: Picture a three-stage pipeline as a horizontal flow. On the left, a cluster of sources — a web server, a router, a database, an application — each emitting raw log lines of different shapes. The arrows converge into Logstash (a box labeled "collect → reprocess → dispatch"), where the ragged lines are straightened into a single consistent format. From Logstash the clean stream flows into Elasticsearch (a grid of nodes, an index), where it becomes searchable. From Elasticsearch an arrow rises to Kibana, where the same data renders as histograms, line graphs, and dashboards. The takeaway: source → Logstash (shape the data) → Elasticsearch (store and search the data) → Kibana (show the data) — one direction, three responsibilities.

16.12.6 Features and Benefits of ELK

• Security: protect your Elasticsearch data in a robust and granular way.
• Automated alerting: get notifications about the changes in your data.
• Monitoring: maintain a pulse on your Elastic stack — keep it firing on all cylinders — to make sure everything is working in line.
• Reporting engine: create and share reports of your Kibana charts and dashboards.
• Graphical visualization: have meaningful relations in your data.
• Machine learning support: since you store and visualize a lot of data, ELK helps you implement machine learning like automated anomaly detection — bringing intelligence into your system to automatically detect anomalies. Even upcoming business trendy features can be decided from the ELK data.

Real-world: since you store those data and visualize them, a lot of data is available — that helps to implement machine-learning-based automated anomaly detection. The stored history becomes the training ground: the same historical data that supports baseline comparisons for alarms also trains models that detect anomalies the thresholds would miss.

Real-world & domain connection: ELK (now part of the Elastic stack, including Beats shippers and Elasticsearch's own alerting) is the most common open source log-and-observability pipeline in industry. Banks use Kibana to visualize stock movements, capacity utilization, and transaction patterns — the professor's stated reason Kibana is the most widely used visualization tool in the banking domain. The same pipeline powering infrastructure monitoring also feeds business dashboards for executives deciding upcoming offerings, and its stored data is the raw material for automated anomaly detection — machine learning applied to the very monitoring data this lecture has been describing all along.

16.13 Knowledge Check

16.13.1 Question 1: What Makes DevOps a Successful Methodology?

16.13.2 Question 2: What Does Not Contribute to the Value Stream?

16.13.3 Question 3: Which Release Pattern Does Not Enable Low-Risk Deployments?

16.13.4 Question 4: What Is Required for Using Jenkins?

16.13.5 Question 5: Which Is Not One of the Three Monitoring Layers?

Visual intuition (the five-question sweep): think of the knowledge check as a quick review map of the course's spine. Question 1 restates the DevOps mission (teams + quality/monitoring + identify-fix-learn); Question 2 draws the boundary of the value stream (create vs maintain); Question 3 tests the release-risk ladder (canary, blue-green, rolling are low-risk; big bang is not); Question 4 checks the CI/CD prerequisites (repository + build script); Question 5 re-checks the monitoring layers (application, business logic, operating system — not business metrics). The takeaway: each question compresses one core theme of the course into one decision.

16.14 Git Branching and Release Tagging

16.14.1 The GitHub Assignment Scenario

A question about an assignment involving GitHub led to a useful walkthrough of a common branching strategy. The goal of that assignment: create branches — one production release branch (the master branch) plus subsequent branches where the team works, then push and merge the completed changes into the master branch.

This is the scenario of creating multiple branches and pushing changes to a particular branch — the essence of a git flow with a production/master branch, working branches, merging, and release tagging.

Visual intuition: Picture the commit graph: a straight horizontal line (master) with two branches arcing above it — one branch rejoins the line, then the other. At the point where the first branch rejoins, a small label hangs on the line: v1.1. The graph is the story of the release: the horizontal line is the history that went to production; the arcs are the parallel work; the label is the moment the world's baseline changed.

Real-world & domain connection: This master/working-branch + tag pattern is the foundation of professional release management: GitHub's release mechanism, trunk-based development with short-lived feature branches, and versioning schemes (semantic versioning tags like v1.1.0) all build on it. In CI/CD pipelines — the practical side of this course — every pipeline stage checks out a tagged commit or a merged branch, and rollbacks pick the previous tag. The Jenkins prerequisites from the knowledge check connect here: the repository and the build script both live in this branch structure, and the release tags are what deployment automation deploys.

Exam Guidance Summary

• Continuous monitoring mindset: understand proactive vs reactive maintenance — monitoring should catch problems before the end user notices them (the ticket-then-debug workflow is the traditional reactive way). This is the single mindset the whole lecture is built on.
• Definition questions: monitoring is a process (not a tool or system) to observe and record state changes and data flow; state change via direct measurement or logs; data flow captured by logging request and response data, internal and external. Know the two ways to express a state change and the two kinds of data flow.
• Five goals of monitoring: failure detection, performance problems, workload categorization/capacity planning, user reaction to business offerings, intrusion detection. Know the data-source table: all goals take application + infrastructure data except user reaction, which is application-only — the one-row exception is a favorite trap.
• Fundamental items: input, resources (hardware: CPU/memory/disk/network; software: queues, thread pools, configuration specifications), output (transactions, business outcomes). The input → resources → output loop.
• Failure detection: total vs partial failure (cable pin example — total = no data flow, easy; partial = changing response times, hard), dependency software failure (Tomcat) vs software misconfiguration (JRE 7 → 8), and the three detection ways (external monitoring software health check, in-system special agent, self-detection).
• Performance metrics: latency (time from initiation to completion, from user request to satisfying the request — $L=t_{\text{end}}-t_{\text{start}}$), throughput (operations of a type per unit of time — $T=\frac{N_{\text{ops}}}{\Delta t}$, the 50 → 45 report generation example), utilization (relative use of a resource — $U=\frac{r_{\text{used}}}{r_{\text{total}}}\times 100\%$, the 80 percent threshold, degradation at 90-100 percent).
• Capacity planning: long-term (human involvement, days/weeks/months/years, XYZ paper-distribution example: 400 target vs 600 observed in 15 days → 1,500-user capacity order) vs short-term (automatic, minutes/hours, VM creation/shutdown for cost, Amazon 100 → 50 VMs at night saving 50 percent cost; pay-per-use defined by NIST). The presence or absence of human interaction distinguishes the two.
• User interaction monitoring: real user monitoring vs synthetic monitoring (script-driven artificial users for capacity testing and UAT). Three observations: request latency, reliability under load, UI modification.
• Intrusion detection: role-based activity monitoring (admin report 100 times at 3 p.m.), network traffic anomaly detection (Pune vs Amsterdam/Japan — verify rather than lock), historical data + attack-pattern libraries, passive vs active intruders.
• How to monitor: health checks/heartbeats (bit 1 healthy, 0 unhealthy — or vice versa by convention), agent-based vs agent-less (decision scenarios covered in the upcoming session), monitoring + configuration management working hand in hand; three main features: visualization, alarms/alerts, data storage.
• Operation activities: configuration conformance to desired state (Chef, Puppet, SaltStack agent-based; Ansible agent-less), time-series collection with the three challenges (by time, by context, volume), logs (sources, uses, the four log-writing best practices: consistent format, explanation, context, screening), graphing/displaying, alarms and alerts (alert = informs, alarm = requires action; 80 percent alert / 85 percent alarm; temperature rise = alert, fire = alarm; issue → ticket → alarm → alerts → diagnosis → remediation).
• DevOps challenges: continuous change (automate alarm/alert/threshold configs, auto-register/deregister servers), bottom-up vs top-down (hardware failures rise to the application level — bottom-up resolves earlier; combine both), microservices (loosely coupled services — intelligent monitoring system), large data volumes (finer 10-minute collection during upgrades, 30-minute after 15 days).
• ELK: Elasticsearch (search engine, NoSQL on Lucene, RESTful, JSON key-value, aggregation), Logstash (server-side pipeline: collection, reprocessing, dispatching), Kibana (visualization: histograms, line graphs, pie charts, sunburst), architecture flow (logs → Logstash → pre-process → Elasticsearch → search → Kibana), features (security, automated alerting, monitoring, reporting engine, ML anomaly detection). ELK is on the syllabus at the architecture level.
• Knowledge check style questions: DevOps success factors (all of the above), value stream (errors/incidents/fixes do not contribute — they are post-implementation), release patterns (big bang does not enable low-risk deployments; canary, blue-green, rolling do), Jenkins requirements (both: Git repo + build script), monitoring layers (business metrics is not one of the three: application, business logic, operating system).
• Practical/exam logistics: the upcoming quiz follows the tutorial session; quiz 4 syllabus was published on the learning portal. ELK and configuration management are not part of the practical sessions — the practical is CI/CD and deployment, including containerized deployment with Docker and Kubernetes. An ELK getting-started document will be shared for self-study.

Key Industry Applications

• ELK stack (Elasticsearch, Logstash, Kibana): the most common open source monitoring/logging solution in industry; Kibana is the most widely used visualization tool in the banking domain — banks use its dashboards for stock movements, capacity utilization, and business reporting.
• Amazon AWS: on-demand infrastructure where hardware monitoring is the cloud provider's responsibility (the total/partial hardware failure burden moves to the data center provider); Amazon CloudWatch as the monitoring service inside the cloud.
• Amazon e-commerce (India region): short-term capacity planning in practice — shutting down 50 of 100 VMs during 1 a.m.–5 a.m. saves 50 percent of the cost, driven by monitored usage patterns and configuration management activation rules.
• Configuration management tools: Chef, Puppet, and SaltStack (agent-based) and Ansible (agent-less) keep systems at the desired state, detecting drift and converging configurations.
• Hadoop / big data: storage platforms for the large volume of monitoring data — the answer to the volume challenge of time series collection.
• Kafka: also used by many organizations to collect and monitor events — suitable for basic monitoring, though not every service of a strong monitoring system (discussed in closing Q&A).
• Sensu, Nagios: other monitoring tools in the DevOps tooling landscape — Nagios for its plug-in pool and alerting, Sensu for cloud-scale extensibility.
• Tomcat, JRE: concrete dependency examples used for software failure scenarios — Tomcat for dependency failure, the JRE 7 → 8 upgrade for misconfiguration.
• Docker and Kubernetes: upcoming tutorial/practical topic — deploying microservices on a container platform, where the health-check heartbeat bit of this lecture is how orchestration probes containers.
• Jenkins with Git and Maven: the practical CI/CD combination (source repository + build script checked in) — the two prerequisites the knowledge check tested.