Malware Detection and Classification

# Malware Detection and Classification

8.1 Malware: Definition, Types, and Evolution

Malware — short for malicious software — is any program inserted into a system, usually covertly, with the intent of compromising the confidentiality, integrity, or availability of the victim's data, applications, or operating system. At its simplest, malware might just consume resources or spam a network with noise. At its most advanced, it can lock down an entire organization's infrastructure through ransomware, with threat actors demanding payment and offering no guarantee of recovery.

The history of malware stretches back to the 1970s and 1980s, when simple viruses spread via floppy disks. A landmark event was the ILOVEYOU worm, created by a student in the Philippines. His professors rejected the project as illegal, but he released it anyway, causing billions of dollars in damage worldwide. It took only three days for Melissa (an earlier email worm) to infect over 100,000 computers, compared to the months it took the Brain virus to infect a few thousand a decade before. Today's landscape includes advanced persistent threats (APTs), ransomware epidemics, and fileless malware that leaves minimal forensic traces.

The key takeaway from this evolution is that signature-based detection alone is not sufficient. Machine learning-based detection of variants and zero-day threats is a necessity.

8.1.1 Malware Classification by Type

Each malware type operates differently and requires distinct detection strategies. Understanding these types is essential because the detection approach must match the malware's behavior — a strategy that catches worms will miss Trojans.

Virus — An early-generation malware that replicates itself but does not cause significant harm on its own. It needs a host file or boot sector to propagate. Think of it as a biological virus that cannot survive outside a host. A computer virus has three parts: an infection mechanism (how it spreads), a trigger (the condition that activates the payload), and a payload (what it actually does beyond spreading). During its lifetime, a virus goes through four phases: dormant (idle, waiting for a trigger), propagation (placing copies of itself into other programs), triggering (activated by some system event), and execution (performing its intended function). Modern operating systems with tighter access controls have made traditional executable viruses harder to spread, leading to the rise of macro viruses that exploit active content in documents like Microsoft Word or Excel files.

Worm — A network-aware variant of a virus. Unlike a virus, a worm can spread across networks without requiring a host file. It self-replicates and propagates autonomously by exploiting software vulnerabilities in remotely accessible network services. Worms use several access mechanisms to spread: email or instant messenger, file sharing via removable media, remote execution capabilities, remote file access, and remote login. The first known worm implementation was done at Xerox Palo Alto Labs in the early 1980s — it was non-malicious, simply searching for idle systems to run a computationally intensive task. Worms follow a propagation model similar to epidemic models in biology, where the number of newly infected hosts is proportional to the product of currently infected hosts and susceptible hosts.

Trojan — Named after the Trojan horse of Greek mythology, this malware disguises itself as legitimate software. A calculator app or a standard Windows utility might actually contain a backdoor. The critical insight is that you can only identify a Trojan through behavior analysis — static analysis alone is insufficient. You examine what the program does, not just what it claims to be. Trojans fit one of three models: (1) continuing to perform the original function while additionally performing a separate malicious activity, (2) modifying the original function to perform or disguise malicious activity, or (3) completely replacing the original function. Unlike worms, Trojans do not self-replicate.

Ransomware — Malware that encrypts a victim's files and demands payment for the decryption key. Detection relies on identifying suspicious API call patterns, particularly calls to cryptographic functions. Ransomware offers a uniquely straightforward cash-out mechanism for attackers — customizable ransomware can be purchased from underground marketplaces for tens of dollars, and at a cost of about 180 USD per thousand successful installations in affluent regions, with even a 10 percent payment rate at 50 USD per victim, the perpetrator's expected earnings exceed 25 times the initial investment.

Rootkit — Malware designed to hide its own presence. Even when installed, a rootkit can remain invisible to the operating system and standard security tools. Rootkits can operate at different levels: user mode (intercepting API calls and modifying returned results), kernel mode (intercepting native API calls inside the operating system kernel), or even virtual machine mode (installing a lightweight hypervisor below the OS). The arms race between rootkit authors and defenders is a continuing "layer-below" battle — as defenders detect rootkits at one layer, attackers move to deeper, harder-to-detect layers.

Botnet — Malware controlled through a command-and-control (C2) network. The infected machines (called bots) receive instructions from a remote server. This concept connects directly to the intrusion kill chain's command-and-control phase discussed in earlier lectures. Bots are distinguishable from worms by their remote control facility — while a worm propagates and activates itself, a bot is controlled by a C&C server network. Early botnets used IRC servers for control; more recent ones use covert HTTP channels or peer-to-peer protocols to avoid single points of failure.

8.1.2 Ransomware Attack Pattern and Detection Opportunities

A typical ransomware attack follows a predictable four-stage pattern:

1. Infiltration — Often through a phishing email containing a malicious attachment
2. Privilege escalation — The malware exploits vulnerabilities to gain higher access rights
3. Rapid encryption — The file system is encrypted at high speed
4. Ransom display — A note appears demanding payment

Each stage creates distinct detection opportunities:

Static detection — Suspicious email attachments can be flagged before they are opened. File hashes, signatures, and structural analysis of attachments provide early warning. This is the fastest detection layer — it does not require executing the file.

Dynamic detection — Unusual API call sequences serve as strong indicators. If a program that normally never calls cryptographic APIs suddenly starts calling them repeatedly, that is anomalous behavior. This is the domain of behavioral analysis running in a sandboxed environment.

Behavioral detection — Mass file modifications trigger alerts through file integrity monitoring. Unix and Linux systems have built-in tools for this purpose. When files are being modified at an unusual rate, something is wrong. This layer catches ransomware that has already begun encrypting but has not yet completed.

Network detection — Command-and-control communication, beaconing patterns, and unusual outbound traffic all indicate potential C2 activity. Ransomware must communicate with its operator to receive the encryption key or to report success — this network activity is a detection opportunity.

8.2 Static versus Dynamic Analysis

Static analysis examines a file without executing it. You look at file structure, file size, imported libraries, section headers, strings, and other metadata. The advantage is speed — you do not need to run the software, so analysis is fast, often completing in milliseconds per sample. The disadvantage is that malware authors can obfuscate static features through packing, encryption, and other techniques.

Dynamic analysis involves actually executing the file in a controlled environment (a sandbox) and observing its runtime behavior. You monitor what files it touches, what registries it modifies, what network connections it makes, what processes it spawns, and what API calls it sequences. The advantage is that you see what the malware actually does. The disadvantage is that it requires time, resources, and careful isolation.

8.2.1 PE (Portable Executable) Structure

The PE header provides insights into whether a file is legitimate or has been tampered with. By examining the PE structure, you can determine what the file is attempting to do. This is a core component of static analysis for Windows malware. Every Windows executable (.exe, .dll) follows this format, making PE analysis a universal tool for examining suspicious files on Windows systems.

The PE file format is analogous to ELF (Executable and Linkable Format) on Linux/Unix systems and APK (Android Package Kit) on Android. Each format has its own header structure, but the principle is the same: the header tells the operating system how to load and execute the program.

Key features extracted from PE analysis include:

• Entry point — Where execution begins. The address of the first instruction the CPU will execute when the program starts. In legitimate software, the entry point typically points to standard initialization code. In malware, it may point to injected or obfuscated code.
• Section entropy — How random each section is. Entropy measures the unpredictability of byte values in a section. Typical benign files have entropy around 5.0 to 6.0 (on a scale where maximum randomness is 8.0). Values significantly higher — say 7.0 or above — suggest the section is packed or encrypted, because random-looking data is a hallmark of encryption.
• Import/export tables — What functions the file uses from external libraries. The import table lists all external functions the program calls (e.g., __CreateFile, WriteFile, InternetOpenUrl__). If a simple calculator program imports cryptographic functions and network APIs, that is suspicious.
• Section tables — How the file is organized. A PE file is divided into sections (like __.text for code, .data for initialized data, .rsrc__ for resources). The number, names, and sizes of sections can reveal tampering.

When entropy is high, it is a strong indicator that the file is packed. Packing is a technique malware authors use to evade signature-based detection by compressing or encrypting portions of the code. The packed file looks different each time (different encryption key), so signature matching fails. However, the high entropy itself becomes a detectable feature — a paradox for the malware author.

8.2.2 Sandbox Requirements for Dynamic Analysis

The sandbox used for dynamic analysis must be completely isolated from production networks. Malware has evolved to detect virtual environments — for example, it can check for virtual network interfaces and, upon detecting them, remain quiet and refuse to exhibit its true behavior. This is an arms race: as sandboxes get better at hiding, malware gets better at detecting.

A simple program can detect whether it is running inside a virtual machine by checking what network interfaces exist on the system. If virtual network interfaces (like __vboxnet0 for VirtualBox or vmnet__ for VMware) are present, the program is likely running inside a VM. Malware authors use this technique to detect sandboxes and remain quiet during analysis.

Several sandbox tools exist for different platforms and use cases:

• GeoBox — A free sandbox for running malware locally. Useful for quick manual analysis without cloud dependencies.
• VirusTotal sandboxes — Multiple sandboxes (Zenbox, CAPE, etc.) integrated into the VirusTotal platform. When you upload a file, VirusTotal runs it in several different sandbox environments and reports behavioral data from each.
• Livan — An open-source Linux malware sandbox written in Python, maintained by a researcher who presents at Black Hat conferences. Specialized for Linux malware analysis.

8.3 N-gram Analysis for Malware Detection

N-gram analysis was previously discussed in the context of feature engineering for URL analysis. The same technique applies to malware detection, but at a different level — byte level or opcode level. The core idea is identical: break a sequence into overlapping chunks of fixed length, then use the frequency or presence of those chunks as features for machine learning.

An n-gram is a contiguous sequence of n items from a given sequence. A unigram (1-gram) is a single item, a bigram (2-gram) is two consecutive items, and a trigram (3-gram) is three consecutive items. In the URL analysis context, these were character sequences. In malware analysis, they can be bytes (raw binary values) or opcodes (processor-level instructions).

8.3.1 Byte-level versus Opcode-level N-grams

Byte n-grams operate on raw bits and bytes. They are faster to extract because you work directly at the byte level without needing to understand the programming language or disassemble the code. Byte n-grams capture local code patterns and can detect even subtle modifications — as little as a 5% change in the file. They are useful for identifying malware families, where a new variant shares most of its code with a known malware but has minor differences.

The trade-off is that byte n-grams provide limited semantic insight. You can detect that something changed, but you cannot easily interpret what the code is doing. It is like detecting that a book has been edited by comparing word frequencies, without understanding what the book is about.

Opcode n-grams operate on processor-level instructions. To extract these, you must first disassemble the file — technically, disassembly rather than full reverse engineering — and then analyze the instruction sequences. Opcode n-grams provide much more meaningful semantic information. You can identify specific operations, such as AES encryption instructions, which directly reveal the malware's intent. They are robust to register changes and allow you to interpret what the file is actually doing.

The trade-off is that opcode n-grams require disassembly, which is computationally more expensive. Disassembly tools like IDA Pro, Radare2, or Capstone are needed to convert raw bytes into human-readable assembly instructions.

8.3.2 Why N-grams Work for Malware Families

When a family of malware exists — similar to how coronavirus had many variants — the overall behavior is similar but there are minor variants in how that behavior is achieved. N-gram analysis can identify these similarities by comparing the byte or opcode patterns between files. If two files share significant n-gram overlap, they likely belong to the same malware family.

The Conficker worm is a famous example. Even though there are many variations of Conficker, each with different code, authors, and behavior, certain characteristics cause them to be attributed to the same malware family — they all exploit Windows OS vulnerabilities and engage in dictionary attacks to crack the administrator account. N-gram analysis captures these shared code fragments across variants.

The analogy to biological virus variants is apt. COVID-19 variants (Alpha, Delta, Omicron) share most of their genetic sequence but differ in specific mutations. Similarly, malware variants within a family share most of their byte or opcode patterns but differ in specific code changes. N-gram analysis detects these shared patterns, enabling family-level classification even when individual variants are new.

8.4 Feature Extraction and ML Pipeline

A typical machine learning pipeline for malware detection combines multiple feature sources into a layered system. The pipeline is not simply passing a file through a single model — it is a staged process where different feature sources contribute different signals.

After feature extraction, algorithms like random forests can be applied, with class weight balancing and cross-validation across malware families. Typical performance ranges from 85% to 95% accuracy, with analysis completing in milliseconds per sample. The choice of algorithm matters less than the quality of features — a well-featured random forest will outperform a poorly-featured deep neural network.

Latency is a practical consideration. If the system is scanning email attachments, the acceptable delay depends on the use case. For web browsing or file downloads, users expect near-instantaneous results — the Gmail warning that appears when downloading a file is an example of real-time malware analysis that must complete in under a second. For endpoint protection scanning files on disk, a few seconds is acceptable. For forensic analysis of suspicious samples, minutes or even hours are tolerable.

The key recommendation is to use static analysis as a fast filter. When 10,000 files arrive, static analysis quickly isolates suspicious ones for deeper dynamic analysis. Multi-layered approaches consistently outperform single-method solutions. This is a core principle in security engineering: defense in depth.

8.4.1 API Call Sequence Analysis

API call sequences reveal what a program is attempting to do at runtime. Different malware types exhibit distinct call patterns:

• Ransomware — Calls to cryptographic functions (__CryptEncrypt, CryptGenKey), file operations (CreateFile, WriteFile, DeleteFile), and file movement (MoveFile__)
• Keylogger — Calls to input capture functions (__GetAsyncKeyState, SetWindowsHookEx) and network transmission APIs (HttpSendRequest, send__)
• Data exfiltration — Calls to HTTP POST (HttpSendRequest), file upload, and remote communication APIs
• Backdoor — Calls to accept remote commands (__HttpSendRequest, accept, recv__)

The critical challenge is distinguishing malicious API usage from legitimate usage. BitLocker, for example, calls cryptographic APIs for disk encryption — this is entirely legitimate. The same API calls that indicate ransomware in one context are perfectly normal in another. This is where machine learning-based analysis becomes essential: it learns the patterns and contexts that distinguish malicious from benign behavior.

8.4.2 Network Behavioral Profiling

Instead of analyzing API calls on the host, you can observe network traffic directly. Behavioral indicators include:

• Which network APIs are being called
• Which URLs are being accessed
• Which ports are being used
• Large data file uploads (potential exfiltration)
• TLS protocol versions
• IP addresses being contacted

8.5 Deep Learning for Malware Detection

Deep learning has introduced several powerful approaches to malware detection that complement traditional ML methods. These techniques excel at automatic feature extraction — the neural network learns what features matter, rather than requiring a human expert to engineer them.

8.5.1 CNN-based Malware Visualization

One of the most innovative approaches in malware detection is converting binary files into grayscale images and applying convolutional neural networks (CNNs) for classification. This technique translates a cybersecurity problem into a computer vision problem.

The process works as follows:

1. Take a binary file (Windows executable, shell script, etc.)
2. Convert the raw binary data into a grayscale image with fixed width and height
3. Treat the binary data as pixel values (each byte becomes a pixel intensity from 0 to 255)
4. Visualize the resulting image

Why does this work? Binary files contain underlying patterns. Code sections have distinct textures — compiled code has a different visual signature than data or resources. Packed files create noise in the image because encrypted or compressed data looks random at the byte level. Families of malware produce similar visual patterns — these are called visual signatures. CNNs excel at detecting these visual signatures because they are designed to recognize patterns in images, regardless of their source domain.

The architecture typically consists of multiple convolutional blocks that extract increasingly abstract features from the image. Early layers detect simple patterns (edges, textures); deeper layers detect complex structures (code patterns, packing signatures). The CNN learns to distinguish between benign and malicious files based on their visual signatures.

This approach is particularly powerful because it requires minimal feature engineering — the CNN automatically learns relevant features from the raw binary representation. However, it has limitations: it requires large training datasets, and the learned features are not easily interpretable by humans (unlike n-grams or API calls, which have clear semantic meaning).

8.5.2 LSTM for API Sequence Analysis

Long Short-Term Memory (LSTM) networks are a type of recurrent neural network (RNN) designed to process sequential data. For malware analysis, LSTMs are applied to API call sequences.

The key insight is that order matters in API calls. Any program executes calls in a specific sequence — you cannot call one function before another without changing the program's behavior. LSTMs exploit this sequential nature to detect anomalous patterns. A simple count of API calls loses the ordering information; an LSTM preserves it.

The process involves:

1. Extracting API call sequences from dynamic analysis — each API call is treated as a token in a sequence
2. Feeding these sequences into an LSTM network — the LSTM processes one call at a time, maintaining an internal memory of what it has seen so far
3. The LSTM learns normal call patterns and flags deviations — sequences that do not match the learned distribution are flagged as anomalous

If the LSTM detects a sequence that does not match known patterns, it signals that something suspicious is happening. The LSTM can be trained on sequences from known malware families (for classification) or on sequences from benign programs only (for anomaly detection).

8.5.3 Autoencoders for Zero-Day Detection

Autoencoders provide a powerful approach for detecting zero-day malware — threats that have never been seen before and for which no signatures exist.

The training process works as follows:

1. Collect benign samples from trusted sources — GitHub repositories, official build rooms, vendor-provided authentic files with known hashes
2. Train the autoencoder to learn the normal representation of benign files — the autoencoder compresses the input into a low-dimensional representation (the bottleneck) and then reconstructs it
3. At detection time, pass new files through the autoencoder
4. Measure reconstruction error — if the autoencoder cannot reconstruct the file accurately (high reconstruction error), the file is anomalous and potentially malicious

8.6 Evasion Techniques and Defense Strategies

8.6.1 Malware Evasion Techniques

Malware authors employ several techniques to evade detection, each targeting a different layer of the detection stack:

Code packing and compression — Encrypting or compressing portions of the payload to hide the true code from static analysis tools. The packed file has high entropy (appears random), defeating signature-based scanners. When the packed file is executed, a small stub of code decrypts the real payload in memory. This is why section entropy analysis (discussed in PE structure) is an important feature — high entropy is a signal of packing.

Anti-analysis techniques — Detecting whether the malware is running inside a sandbox or virtual machine. If a virtual environment is detected, the malware stays quiet and does not exhibit its true behavior. Common detection methods include: checking for virtual network interfaces, looking for VMware/VirtualBox registry keys, checking for specific hardware identifiers, and measuring the time taken for certain operations (virtual machines are often slower than bare metal). Some malware will delay execution by several minutes — longer than most sandbox analysis runs — to outwait the analysis period.

Polymorphism — The behavior remains the same, but the code is different each time. Consider how many different ways you can write a "Hello World" program — you can use different variable names, add comments, reorder independent statements, insert NOP (no-operation) instructions. Malware authors apply the same principle — the malicious intent is identical, but the implementation varies. For example, if the goal is to exfiltrate data, there are thousands of different ways to write the code, but the behavior is the same.

The textbook distinction between polymorphism and metamorphism is important: polymorphic malware typically contains two sections — the core logic that performs the infection, and an enveloping section that uses encryption and decryption to hide the infection code. The encryption key changes with each copy, so the encrypted portion looks different each time. Metamorphic malware goes further — it injects, rearranges, reimplements, adds, and removes code in the malware itself. Because the infection logic is not altered between evolution stages in polymorphic malware, it is comparatively easier to detect than metamorphic malware.

Metamorphism — A more advanced technique where the code completely rewrites itself each time it runs. Unlike polymorphism (which just encrypts the payload), metamorphism changes the actual code — reordering instructions, substituting equivalent operations, inserting dead code, and changing control flow. Each generation of metamorphic malware is structurally different from the previous one, making signature-based detection nearly impossible. The only reliable detection method is behavioral analysis — watching what the malware does, not what it looks like.

8.6.2 Defense Strategies

8.7 Exam Guidance Summary

8.7.1 Exam Structure and Format

The exam follows this structure:

• Total marks: 100
• 4 questions × 8 marks each = 32 marks for short reasoning questions
• 4 questions × 12 marks each = 48 marks for applied reasoning questions (definitions will be part of these)
• 2 questions × 5 marks each = 10 marks for extended answers (explain strategy or approach)

Syllabus: All 8 sessions, content from slides.

8.7.2 Topic-wise Study Guidance

What to study from each major topic:

Core concepts (2-3 mark questions expected): Definitions of risk, vulnerability, control, impact, incident, threat actor, CIA triad, attack vectors. Be able to define each term precisely and give one example.

Intrusion kill chain: Know each phase from reconnaissance to actions on objectives. Understand attacker opportunities and defender opportunities at each step. Attack matrices are applications of this concept.

NIST Cybersecurity Framework: Identify, Protect, Detect, Respond, Recover. The analogy: if you are a doctor studying diseases, the intrusion kill chain is your focus. If you are a defender maintaining health, NIST CSF is your framework — the habits that keep you prepared.

Threat intelligence: Threat actors, motivations, DDPs, threat modeling approaches. Expect applied questions like "How do you threat model for ransomware?"

Data collection and preprocessing: Structured versus unstructured data sources, their formats, and use cases. An example question: "You are tasked to build an email spam detection engine. What structured and unstructured data sources would you consider and why?"

Feature engineering: Brute force detection features, malicious login features, whether to use mean or average. Questions may use different examples than those discussed in class.

Algorithm selection: When to use supervised versus unsupervised learning. Consider constraints like available resources, labeled versus unlabeled data, latency requirements, and the specific problem statement.

Anomaly detection: Statistical methods, distance-based methods, density-based methods. Go beyond the slides to understand the nuances.

False positive analysis: The accuracy trap, precision versus recall trade-offs, area under the curve. When false positive rate is high, what matters more — precision or recall?

Malware detection (this lecture): Polymorphic malware detection, malware family identification, n-gram analysis, static versus dynamic analysis trade-offs. Know the difference between polymorphism and metamorphism. Understand why behavioral detection is the primary countermeasure against evasion.

8.8 Key Industry Applications

8.8.1 Security Tools and Platforms

• VirusTotal — Google-acquired service for file analysis, combining dozens of antivirus engines with static and dynamic analysis in multiple sandboxes (Zenbox, CAPE, etc.). It provides both static details (hash, type, DLLs, strings) and behavioral data (registries touched, URLs accessed, network calls made). VirusTotal is a critical resource for both manual analysis and automated threat intelligence.
• GeoBox — Free sandbox for local malware detonation. Useful for analysts who need to run malware in an isolated environment without relying on cloud services.
• ClamAV — Open-source antivirus with signature libraries. Widely used in email gateways and file servers as a first line of defense.
• Livan — Open-source Linux malware sandbox written in Python, maintained by a researcher who presents at Black Hat conferences. Specialized for Linux malware analysis.
• BitLocker — Microsoft disk encryption. Used here as an example of legitimate cryptographic API usage — BitLocker calls the same cryptographic APIs that ransomware uses, but for entirely legitimate purposes. This illustrates the context-dependence challenge in malware detection.
• CrowdStrike, Sentinel One, Trend Micro — Commercial endpoint protection platforms. Used as examples of the vendor diversity challenge — different endpoints in an organization may require different security vendors, making uniform deployment difficult.
• Zscaler/ZIA — Zero-trust security tools that route all traffic through inspection. Represents the network-level detection approach where all traffic is analyzed regardless of source.
• Wireshark — Network traffic analysis tool for observing malware network behavior. Essential for manual network-level analysis of suspicious traffic.
• GitHub Copilot — AI coding assistant. Mentioned as an example context for supply chain security incidents.

8.8.2 Real-world Incident: Supply Chain Attack via GitHub Copilot