Malware Detection: A Real-Life Data Mining Application

This is the last class of the data mining course, and it turns the tables: instead of learning a new algorithm, we apply everything learned so far to one real problem — malware detection, the task of building an antivirus system using data mining techniques. Everything discussed from the very first class through this one is part of the comprehensive exam syllabus. Modules seven and eight were never covered in class, so they are excluded from the comprehensive exam.

The session is built as a complete case study. We start with the problem itself and why it is hard, review the data mining machinery we will reuse (classification, clustering, performance measures), survey how the literature organizes malware detection, then walk through three published works that build a permission-based Android malware detector with machine learning and deep learning. The story ends with a twist: the same five-step pipeline that builds the detector can be turned against it, and the winning strategy is to attack your own system before the bad guys do.

16.1 Malware Detection: The Game

16.1.1 What, Why, and How

The session is organized as what, why, and how. First, understand the current security challenges and the environment: how bad the malware problem is and what makes it hard. Then see why this is a relevant problem: there are gaps in the existing literature and technical challenges in the existing solutions. And finally, look at how we can build a next-generation malware detection system — a next-generation antivirus — using data science, data mining, machine learning, and deep learning, and what the future directions of the domain look like.

In layman's terms, malware detection is the task of building an antivirus system using data mining techniques. The existing literature matters: what solutions have people already built, and where do they fail? And the challenges matter: what makes the problem hard, and how do we check whether a proposed solution is even feasible?

The same mental template — understand the domain, find the gap, build, measure, attack your own solution — carries over to almost any domain where you apply data mining to a real problem. Whether the problem is spam filtering, credit-card fraud, or network intrusion, the sequence is the same: learn the domain first, locate the weakness in current solutions, build a model, measure it honestly, and then try to break your own model before someone else does.

16.1.2 The Zero-Sum Game

Malware detection is a game. It is a two-party game between malware designers on one side and the anti-malware community on the other, and it is a zero-sum game: whatever benefit the malware designer gets is a loss to the anti-malware community, and the other way round. Both sides try to outrun each other.

Today a malware designer launches a malware; the aim of the anti-malware community is to detect it and block it as soon as possible. Whatever malware designers do, the anti-malware community counters. On the other hand, if the anti-malware community comes up with a new detection strategy, the malware designer will build an evasion strategy to evade, fool, or deceive the new solution. The two players push each other forward, and this race never ends.

From the malware designer's perspective there are exactly two jobs. First, write a malicious code — a piece of code that performs some malicious activity. Second, build a mechanism so that the code evades the detection system. The second job exists because detection matters: if the antivirus catches the malware, it blocks the malicious code and the designer's aim fails. So the designer must both write the malicious code and make it invisible to the detector.

16.1.3 The McAfee Prediction and the Endless Game

The session opens with a quote. In 1988, John McAfee — one of the founders of McAfee antivirus, a legendary person in the industry — said that the problem of viruses is temporary and would be solved in the next two years. Fast-forward roughly 35 years to 2023, and we still face challenges from viruses, worms, malware, and so on. The 1988 prediction turned out to be wrong.

Our hypothesis, based on the work discussed here, is that security is an endless zero-sum game between malware developers and the anti-malware community. From the anti-malware community's perspective, the winning move is to think, design, and develop one step ahead: whatever the malware developers are going to do next, the anti-malware community should be able to anticipate it, maybe design a solution for it, and develop that solution before being attacked. The game is a seesaw — sometimes the anti-malware community wins, sometimes the malware designers win — but the party that stays one step ahead stays superior in the game.

16.1.4 The Roadmap of This Session

The plan: a basic introduction; some preliminaries (recapping the data mining machinery we will reuse); a survey of the existing literature in malware detection; three published works on permission-based malware detection using machine learning and deep learning, which are available online and can be searched by title for more detail; and finally conclusions plus future directions. The three works are the source of most of the results shown here.

This is a huge domain. A lot of work has already been done, and a lot more will be done in the next few years, so the session cannot cover everything in one sitting. What it offers is a crust — a working sketch — of how things look when you apply data mining to a real problem: the challenges, the feasibility checks, the different perspectives, and the different performance measures. That crust transfers to almost every other domain where data mining is applied.

Real-world: all of the classical algorithms studied in this course — classification, clustering, association rule mining — have been used by different authors to build malware detection systems, with mixed accuracy and different issues. Searching Google Scholar for "malware detection classification", "malware detection regression", "malware detection clustering", or "malware detection association rule mining" returns tons of papers.

16.2 What Is Malware

16.2.1 Definitions

The term malware comes from malicious software. According to one source, malware is a software program that meets the harmful intent of the malware designer or developer. According to NIST, malware enters the computer system without the user's authorization and performs undesirable actions: stealing information, sending spam emails, compromising the system, crypting infrastructure, and so on. Malware includes viruses, worms, Trojans, spyware, hardware, spamware, bots, rootkits, ransomware, and more.

Two parts of those definitions matter. First, intent: malware is software whose purpose is harmful, not software that merely has a bug — a crashing app is annoying but is not malware. Second, consent: the software acts without the user's authorization, so even a helpful-looking program becomes malware the moment it does things the user never agreed to.

16.2.2 A Short History: From Fun to Profit

Looking at history, malware was initially developed for fun or to show one's technical knowledge. The first malware was designed in 1970, and its name was Creeper; it just displayed a taunting message: "I am creeper, catch me if you can." For the last 30 years or so, people built malware mainly to show their skills or for fun.

Now malware is a profit-driven industry. People can purchase exploit kits at a price, and they can even rent them on a monthly basis — malware as a service. That is how the industry has shaped in the last 30 years. According to Cybersecurity Ventures, global cybercrime damage cost 6 trillion USD in 2021, and it is expected to reach 10.5 trillion USD by 2025.

Real-world: the "I am creeper, catch me if you can" message is historically the first recorded malware, a useful fact about the origins of the domain.

16.2.3 Malware Taxonomy

Malware is categorized based on the property it follows — the defining behavior that separates one family from another. A virus is a piece of code that attaches itself to a benign program and is activated by that benign program (the host program). A worm is a standalone malicious program that runs independently. A Trojan pretends to be useful but performs malicious activity in the background without the user's knowledge or consent. Spyware spies on user activities without the user's knowledge or consent. Ransomware is installed covertly on the victim's computer and executes a cryptovirological attack — in practice, ransomware locks your mobile, laptop, or server, then demands money, and only then unlocks it.

Because these are properties, one or more of them can be combined into hybrid malware, which is deadlier and more powerful in terms of attack functionality. Many real malwares combine multiple properties to be more dangerous in the computing ecosystem.

16.3 Why Android Smartphones

16.3.1 Choosing the Platform

Any malware attacks a computing platform, and many platforms exist today: smartphones, desktops, laptops, servers, IoT infrastructure, and more. This session focuses only on smartphones, and within smartphones, only on the Android operating system. All three published works discussed here are based on Android smartphones.

Why smartphones? Because the number of smartphones sold has been growing exponentially in the last decade — the growth chart is exponential — and according to a Cisco study, roughly 70% of the global population has a mobile phone (one figure given: 69%), which is more than the fraction of people having electricity, a bank account, running water, or cars. Mobile phones have penetrated society very deeply; everyone uses one. If a malware designer exploits a vulnerability in a mobile phone's hardware or software, the impact reaches a huge number of people, and it can have cascading effects: a locked phone for five days, or spam emails sent to thousands of people from a compromised device.

16.3.2 The Operating System Duopoly

Every computing platform — smartphone, IoT, desktop, laptop — uses an operating system for resource management. In 2009, the smartphone operating system market was fractured, with players like iOS, Blackberry, Windows, and others. Fast-forward to 2021, and the smartphone OS market is a duopoly: Android holds roughly 75% market share and iOS roughly 25% (75 to 25). So almost 3 out of every 4 people in the world use an Android-based smartphone. Combined with the exponential growth of smartphones, a vulnerability in Android can impact an enormous number of devices and people.

16.3.3 Android Malware Growth and the Play Store Problem

The number of malware in the Android ecosystem has also grown exponentially in the last decade. In 2008, when Android was launched, the number of malware was in the single digits or low double digits; by 2021 it had reached roughly 50 million. The volume keeps growing.

Many people download applications from the Google Play Store, and even that is not always secure. In 2021, Google blocked roughly 1.2 million malicious or unwanted applications from its Play Store. The failure mode is familiar from the news: an application was accepted and downloaded by millions of people, later found to be malicious in nature, and then discarded — but the people were still using it in the meantime, and it may have created problems for them. This happens because when the application was launched, the Play Store's checking system did not detect the malicious activity in it.

Real-world: the Play Store uses a scanning system for submitted applications; the design of that system (called Bouncer) is not in the public domain. Many antivirus companies use the very strategy discussed here (permission-based detection) inside their products.

16.4 Research Motivation: Why Existing Solutions Are Not Enough

16.4.1 No Single Algorithm Detects All Malware

According to Cohen, there is no single generic algorithm that can detect all possible malware. Current antiviruses are ensembles of several strategies — many strategies working in tandem to detect new as well as old malware: signature-based detection, heuristic-based detection, behavior-based detection, and so on. The proposal of this session is to add one more strategy to that ensemble: machine learning, deep learning, and data science. The ML-based strategy does not replace the existing solutions; it appends to them and makes the overall detection more promising or better.

Real-world: modern commercial antiviruses combine many detection engines — signature, heuristic, behavior, and increasingly ML — because no one engine catches everything on its own.

16.4.2 The Three Vs: Volume, Velocity, Variety

Currently malware thrives on three Vs. Volume: the number of malware we have to handle is huge — 50 million as of 2021. Velocity: the number of new malware arriving each day is huge. Variety: there are many variants of malware, and the variety of new malware is increasing day by day. The anti-malware community needs a solution that can handle volume, velocity, and variety together.

Existing detection systems cannot fully handle the current malware challenges. This is a rat race: the malware designer builds more and more malware and increases its sophistication and complexity, while the anti-malware community wants to detect both old and new malware. Building detection with data science sits in the broad scope of using big data to solve this challenge.

16.5 Preliminaries: The Data Mining Toolkit We Reuse

16.5.1 AI as Applied Statistics

The claim is that data science, data mining, and artificial intelligence can build the next-generation malware detection system. Artificial intelligence is a form of applied statistics that learns from the data and then performs tasks based on the information it has learned. That is exactly what data mining does: you build a model — a classification model — from data, and then use it to make predictions for unseen, unknown data.

16.5.2 Supervised Learning for Malware Detection

There are two types of algorithms: supervised and unsupervised learning. In supervised learning for this domain, the data points are Android applications, and we know each application's class label: malicious or benign. Malicious means a bad application (malware); benign means a good application. We design a function that fits the data with minimum error, using any classification model — a decision tree (DT), random forest (RF), or any other. Once the model is built, we predict the class label for unseen applications. Given a set of Android applications and their labels, a malware detection system predicts whether a test application is malicious or benign. Classification is the main strategy; regression is sometimes used too.

16.5.3 Unsupervised Learning for Malware Detection

In unsupervised learning, we are given unlabeled data — application one, application two, application three, with no class labels — and we build a function that finds pattern and structure in the data. Clustering is the example seen in earlier classes: finding similar data points. Association rule mining is another strategy that can be applied to malware detection. Different authors have investigated all of these strategies — classification, regression, clustering, association rule mining — for malware detection, and tons of papers exist for each.

16.5.4 Performance Measures and Algorithms Used

Once a classification or clustering model is built, performance measures tell us its quality. In classification we saw accuracy, true positive, true negative, false positive, false negative, confusion matrix, AUC, and so on. In clustering we can use mechanisms like the Silhouette score, or the number of clusters in the elbow method.

The classification algorithms used across the three works cover the full spectrum studied in the course: classical machine learning — decision tree, SVM, K-nearest neighbor; bagging-based strategies — bagged decision tree, random forest, extra trees; boosting-based strategies — AdaBoost, gradient boost, extreme boosting; and deep learning — deep neural networks with one layer, three layers, or seven layers, and convolutional neural networks (CNN). All of these were used to build malware detection systems.

16.5.5 Confusion Matrix Terms in the Malware Domain

By convention (not a hard rule), malware is labeled 1 and benign applications are labeled 0. Given that convention:

• True positive (TP): malware that is actually detected as malware by the detection system.
• True negative (TN): a benign Android application that is detected as benign.
• False positive (FP): a benign application that is wrongly detected as malware.
• False negative (FN): a malicious application that is misclassified as benign.

From these four, accuracy is computed as correct predictions divided by all predictions:

$$\text{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}\times 100$$

The result is a percentage. The four cells are the familiar confusion matrix; building a classification model means assigning a class to each sample, after which every prediction falls into one of the four cells.

16.5.6 Domain Knowledge: Which Measure Actually Matters

Here is an important insight: when applying data mining to a real problem, you need knowledge of the classification algorithms AND knowledge of the domain. In the malware detection domain, the most important performance parameter is the false positive — the papers in the malware detection literature that use machine learning and data science are first and foremost trying to minimize the false positive. Why? Think about what happens when an antivirus makes a mistake. The worst thing it can do is detect a genuine benign application as malicious. A Windows DLL — a benign Windows library, essential for the operating system — wrongly classified as malicious gets blocked or deleted, and that can shut down the whole operating system or application, or bring down a stack running on a laptop or desktop. Everything fails. So the worst outcome is a benign application misclassified as malware.

The same principle exists in law: the basic thumb rule of most legal systems is that you can allow 100 prisoners, cheaters, or criminals to escape, but a benign person should not be punished. Law assumes people are good, and the onus is on the community to prove someone is a criminal. Analogously, a benign application should never be blocked because it was wrongly classified as malicious. So the first, most important parameter in the malware detection domain is the false positive rate — minimize it. The second most important parameter is accuracy — maximize it. Then come other performance measures. This is more of a convention than a rule: read 50 papers and the majority try to minimize false positive and increase accuracy.

The ranking also answers the practical question of what to report in a paper: list several measures together, and let the domain decide which one leads. A student raised exactly that question in class.

16.5.7 Fooling Rate: The Metric for the Adversarial Side

Malware detection is a two-party game, and the malware designer has two jobs: write malicious code, and evade the detection system. The second job is measured with the fooling rate (also called the evasion rate). The idea: a malware designer takes a malicious application $M$ and modifies it by adding a small change $\Delta$, producing a modified malicious application $M^{\prime }=M+\Delta$. The property of $M^{\prime }$ is that it can fool or evade the detection system. Initially the designer's malware was detected; adding some junk to the application makes it undetectable.

Formally, the fooling rate is the percentage of malicious applications successfully converted into adversarial applications — applications that can fool the detection system. With $F$ the detection model and $M^{\prime }=M+\Delta$ the modified sample:

$$\text{Fooling rate}=\frac{\text{number of malicious apps }M\text{ with }F(M^{\prime })=\text{benign}}{\text{total malicious apps tested}}\times 100$$

The numerator counts only the samples that actually fool the detector, and the denominator is the whole test batch — so the rate lies between 0% (nothing fools the detector) and 100% (everything fools it). The percentage form matches how the same measure is reported later in the session: one added permission fools about 25% of malicious applications, and ten additions fool nearly all of them (section 16.8.6).

16.6 The Malware Detection Landscape

16.6.1 How the Literature Organizes the Domain

Malware detection engines can be organized by platform: Android, Windows, iOS, Linux, and so on. Malware is studied based on its taxonomy — what sort of properties it has — and there are two generations. First-generation malware contains only the malicious code and no evasion strategy: early malware designers wrote the malicious code and stopped there. Second-generation malware contains both the malicious code and an evasion strategy — encrypting the malware, developing polymorphism, metamorphism, and similar tricks to evade the detection system. In this session, the detection system targets second-generation malware; there exist systems that detect only specific taxonomies, like only viruses or only worms.

16.6.2 Static, Dynamic, and Hybrid Analysis

From the anti-malware community's perspective, malware is analyzed via static analysis or dynamic analysis. In static analysis, we look into the code to find features that can separate malicious and benign files. A classic static feature is permissions: every Android application asks for permissions (WiFi access, SMS, gallery, and other phone resources), and those permissions can be extracted by looking at the code itself. In dynamic analysis, we run the Android application in a controlled environment — a sandbox — and analyze the log to find features that separate malicious and benign applications. A classic dynamic feature is system calls: whenever an Android application wants to do anything important, it makes a system call; we run the app in a controlled environment, log the system calls in the background (tools like strace can be used), and analyze the log to build a detection system.

Hybrid analysis uses both: static and dynamic features in tandem — two static plus one dynamic feature, or any other combination. Searching "malware detection hybrid analysis" yields around 50 papers; the technique has been used extensively.

16.6.3 Detection Engines: Signature-Based

Traditionally, malware detection systems were based on signature, heuristic, and behavior; this session adds machine learning and deep learning as a next-generation engine. Signature-based detection is a traditional system and is still the backbone of any antivirus. You can visualize it as a hashing strategy: take a malicious code, generate a hash out of it, and store that hash in a database; the hash value is used again and again to detect the malware. Signatures are manually generated, decided, and maintained by domain experts — a human-driven process — and that becomes the bottleneck. With 50 million malware today (possibly 100 million in five years), how many signatures can humans generate? The human-driven solution does not scale.

There are two painful numbers from one study: the typical time window between malware release and its detection or signature generation is roughly 54 days (the professor also gave the figure as about 50 days at one point) — two months in which the malware can do a lot of damage — and 15% of samples are still undetected after 180 days, which is 6 months. Both phrasings point to the same painful reality: roughly two months of undetected activity before a signature exists. A lot of malware roams the internet freely, undetected by any antivirus, and can end up on your mobile device.

The fundamental problem with the signature-based strategy is that it is reactive: you can only build a signature after the malware is launched, because the signature is built from the malware. It is not proactive, and it does not protect against zero-day attacks.

16.6.4 Detection Engines: Heuristic-Based

In heuristic-based detection, domain experts write rules and patterns that separate malicious and benign files. A simple example from everyday computing: on your laptop or mobile phone, all ports are closed except a few; if an application wants to access a certain port, it must open that port, and you are advised to close it when the job is done. Why close all ports? Because you do not want any application communicating through a port that is not monitored, and putting a monitoring mechanism on all ports is costly. So experts write rules like this to separate malicious and benign activities.

The rules must be general enough to detect malware of the same family, but must not raise false alarms. Heuristic detection is proactive — it can catch families and patterns rather than exact copies — but it needs a lot of domain knowledge to write the rules, and it has the false-positive problem: a wrong alarm can have a cascading effect and block genuine applications. That is why heuristic detection is used in a limited fashion.

16.6.5 How a Typical Antivirus Works

The diagram on the screen shows how a typical antivirus looks: two parts, a client side (your laptop) and a server side (the antivirus infrastructure, say the infrastructure of a large antivirus company). On the server, a crawler crawls the internet for malicious samples and tries to find new ones. Once samples are found, a domain expert looks into them and tries to generate signatures to detect them. Those signatures are stored in the server database and synced to the client side — this is what happens when you update your virus definitions, which happens almost every day: new signatures get copied from the server to your local machine.

Then the antivirus runs: it takes a file (a test file), generates a signature of it, and matches it against the big signature database. If it matches, the test file is malicious; otherwise it is benign.

16.7 Building a Detection System: The Five-Step Data Mining Pipeline

16.7.1 The Five Steps

Machine learning and deep learning can also build a malware detection system, and it is a five-step process — the same process you have seen throughout this data mining course:

The order is fixed on purpose: you cannot extract features before you have data, you cannot analyze features you have not extracted, and you cannot judge a model you have not built. Step 3 is the one students often skip — understanding the data before modeling is what turns a model into an insight machine, as the EDA findings in section 16.7.3 show.

16.7.2 Android Permissions as the Feature Vector

This session uses static analysis and extracts Android permissions. The claim: by looking at the permission usage of an Android application, we can predict whether the application is malicious or benign. The feature vector looks like a CSV: columns are permission one, permission two, permission three, and so on; rows are application one, application two, application three, and so on. If application one uses permission P1, the cell is marked 1; if it does not use permission P2, the cell is 0; and so on. This is a binary feature vector.

The data types connect back to the course: we saw nominal, ratio, interval, ordinal, and binary attributes. Permission usage is a binary attribute. If we extracted system calls instead, that would be a ratio-type attribute.

16.7.3 Exploratory Data Analysis Findings

Once the feature vector is built, we study how it looks: find important features, build a correlation matrix, and check for null values and noise. Two findings from the EDA on this data: ACCESS_NETWORK_STATE was heavily used by benign applications but less used by malicious ones, and RECEIVE_SMS was heavily used by benign applications. These insights tell us which permissions separate the classes.

A correlation grid added another finding: does a malicious application use two permissions in tandem to perform a malicious activity? Yes — READ_SMS was used in tandem with WRITE_SMS, and SEND_SMS was used in tandem with RECEIVE_SMS to perform malicious activity. Building the classification model is good, but EDA like this gives insight into the data: permissions can be used in pairs to perform malicious activity, and some permissions are heavily used by one side or the other. This knowledge helps design the detection system.

16.7.4 Dimensionality Reduction: Escaping the Curse

The three published works also reduced the number of features — dimensionality reduction, which was discussed in the course (PCA, attribute subset selection, forward selection, backward elimination). Four strategies were used: the original data (no reduction, roughly 200 permissions); a variance threshold, where the variance of each permission is computed and the permission is kept only if its variance is above a threshold; PCA; and autoencoders — a deep learning strategy for feature reduction — with one layer and with three layers. From 200 features, the reduced sets were around 16 features for the variance threshold, and 16 to 64 features for PCA and the autoencoders, depending on the latent size chosen.

Why reduce? Because 200 features invite the curse of dimensionality: as the number of features grows, the model performs poorly.

16.7.5 The Research Questions and Problem Definition

The work started with three research questions:

1. Do malware designers use specific Android permissions to perform malicious activity? That is, by analyzing permission usage, can we predict whether an application is malicious or benign?
2. Can we reduce the Android permissions to come up with an efficient malware detection system? The Android system defines roughly 200 permissions, and a 200-feature vector suffers from the curse of dimensionality.
3. Do machine learning models take less computational resources than deep learning models in malware detection?

The problem definition: a dataset $D$ contains Android applications and has two subsets, $M$ and $B$, where $M$ is the subset of malicious applications and $B$ is the subset of benign applications. We assume the number of malicious and benign applications is almost the same, so we do not have to worry about class imbalance. The dataset $D=\{(ap{p}_i,clas{s}_i)\}$ is a labeled dataset, where $ap{p}_i$ is a particular Android application and $clas{s}_i$ is its class label: 0 for benign, 1 for malicious (a convention followed in the work). Features are extracted using static, dynamic, or hybrid analysis (static here: Android permissions). Then a malware detection model $F$ is built with machine learning or deep learning algorithms, and its performance is measured with accuracy, FP, TP, AUC, and the other measures.

16.7.6 The Datasets: Drebin, Play Store, and VirusTotal

For malicious applications, the work used the Drebin dataset, built by a research group and one of the benchmark datasets in the field — it contains roughly 5,500 malicious Android applications from various malware families and has more than 2,000 citations. The dataset was introduced by Arp et al. in their 2014 paper on explainable detection of Android malware, and it is widely used as the standard malicious-app benchmark.

For benign applications, the team wrote a parser that scans and downloads Android applications from Google Play — roughly 8,000 applications were downloaded. Because even the Play Store is not fully secure, the applications were checked with VirusTotal.com: a service owned by Google that is an ensemble of the 50 to 60 most popular antiviruses, with an API where you can upload any Android application and the antiviruses report whether it is malicious or benign. An application was labeled benign only when all the antiviruses declared it benign — a conservative labeling strategy. Applications that did not pass were discarded. The final dataset had roughly 5,500 benign and roughly 5,500 malicious applications.

16.7.7 Extracting Permissions with APKtool

Static analysis was performed on each of the roughly 11,000 applications: APKtool — a third-party tool, freely available — disassembles an Android application, and a parser then extracts the permission usage for that application. Roughly 200 system-defined Android permissions were found, and the feature vector had roughly 200 columns with one row per application.

16.7.8 The Results Table and Key Insights

After feature reduction, any classification algorithm could be used — from DT to SVM to RF to AdaBoost to deep learning. The results table has, per row, the feature reduction strategy (column 1: original data, variance threshold, PCA, autoencoder with one layer, autoencoder with three layers), the classification algorithm (column 2: DT, KNN, SVM, RF, AdaBoost, DNN with two layers, DNN with four layers, DNN with seven layers), and then the accuracy, true positive rate, AUC, train time, and test time.

The table's headline rows — the best combination found:

The other rows (DT, KNN, SVM, AdaBoost, and the DNNs with two, four, and seven layers) were evaluated the same way across all five reduction strategies; the deep learning rows consistently took the most train and test time.

Key insights from the results:

• The highest accuracy was achieved by the random forest model: roughly 94%. In plain terms: give me 100 test files and I will predict correctly for 94 of them — a good number. The AUC was also high, around 98, meaning the model could separate both the malicious and the benign classes well.
• Reducing the features from 197 to 16 (the variance threshold result) dropped accuracy by less than 1% — from 94 to 93.3 — while saving a lot of train and test time. Computationally this is much more effective: less train and test time for almost the same accuracy.
• Deep learning models took a lot of time and computational resources compared to their machine learning counterparts.

A second student asked whether the drop is forced by the reduction itself.

One more student asked a practical question about the course assignment.

16.8 Adversarial Attacks and the Proactive Arms Race

16.8.1 The Framework Flips: The Bad Guy Has the Same Tools

The five-step framework (data collection, feature extraction, EDA, building a classification model, analyzing the result) builds the antivirus — but all these steps are also available to the bad guys. Malware designers already have data, they can extract features, they can do EDA, and they can build their own detection model to understand yours. They might misuse this knowledge to fool the detection system. So once you have built a malware detection system, an adversary (the malware designer) attacks your ML-based model to fool it. Analyzing the attack and developing countermeasures afterwards is called the reactive arms race. The work discussed here follows the opposite approach: the proactive arms race — we build a machine learning model, then attack our own system first, evaluate the impact, and develop countermeasures before deployment. This falls under a broad area called threat modeling of adversarial attacks.

16.8.2 Threat Modeling: Goal, Knowledge, Capability

Threat modeling is defined based on the adversary's goal against the target system. The adversary is the malware designer; the target system is the antivirus. The goals can be: integrity violation — fooling the detection system; breaking the availability of the antivirus; or stealing information from the antivirus — a privacy violation.

Threat modeling is also defined based on the adversary's knowledge about the detection system. The relevant parameters: the training data, the test data, the feature vector, and the classification model (the classification function). If all these parameters are known to the adversary, it is a white-box scenario; if none are known, it is black-box; if only a few are known — say the training set and some features — it is gray-box.

And threat modeling is defined based on the adversary's capability: evasion attack or poisoning attack. If the adversary modifies a test sample, it is an evasion attack; if the training sample is modified, it is a poisoning attack. Then there are perturbations — what sort of modifications are being made.

16.8.3 The New Problem Definition: The G Function

The problem definition now changes. Earlier we found that a malware detection system can be built with machine learning and deep learning, with 94% accuracy. Now the bad guy wants to fool this model and reduce its performance. The adversary generates a function $G$ that takes malicious Android applications, adds a small change $\Delta$, and converts them into an adversarial application. The property of the adversarial application is that it is forcefully misclassified as benign by the detection system:

$$M^{\prime }=G(M,\Delta )\text{where }{M}^{\prime }\text{ is malicious but }F(M^{\prime })=\text{benign}$$

Before the transformation, the detection model $F$ correctly classified the malicious application as malicious. After the transformation, the same model $F$ misclassifies the variant as benign — the class label has changed. The adversary can transform many malicious applications into adversarial applications and drastically reduce the classification model's performance. On the good-guy side, an adversary-aware defender foresees these attacks and develops countermeasures to defend against them — the proactive strategy: behave like an adversary, attack our own system, find the vulnerabilities, and fix them before deployment at the customer end.

16.8.4 The Specific Attack Configuration

The work takes a subset of the threat-modeling table: an integrity attack — take a malicious application, add delta, convert it into an adversarial application; a white-box scenario (the system was also built for gray-box); an evasion attack — the sample is modified before being presented; a targeted attack — targeting the malware family (two families exist here, malware and benign; the attack targets the malware family only); a false-negative attack — the aim is to increase the false negative rate; and perturbations — minimize the number of modifications, and every modification must be syntactically, semantically, and behaviorally possible.

16.8.5 The Proactive Framework and the Q-Learning Agent

The new framework keeps the first three steps as they were: the same datasets (Drebin and Google Play, roughly 11,000 applications), the same features (Android permissions), and the same classification models developed earlier. Now we behave like an adversary and build a proactive system: an agent takes a malicious application and converts it into an adversarial application whose property is that it is actually malicious but lies in the benign zone. With defense, the detection system should classify these adversarial applications back into the malicious zone — that is the aim.

The agent was built with Q-learning, a reinforcement learning strategy; other published works used deep Q-networks (DQN) and actor-critic strategies.

16.8.6 Fooling Rate Results: 94% to 50%

The graph shows the fooling rate on the Y axis from 0 to 1: zero means none of the Android applications were successfully converted into an adversarial application; one means all of them were. The X axis shows the number of perturbations: one, two, three, five, ten — one permission added, two added, three added, five added, ten added. On the right, all the classification models are listed.

• With only one perturbation, roughly 25% of the malicious applications were converted into adversarial applications.
• With ten modifications, essentially all the malicious applications were converted into adversarial applications that could fool any detection model — from DT to DNN.
• The impact: the performance of 94% was reduced to 50%. The DT model that was getting 90 to 95% accuracy was reduced to 50%, and this happened across almost all the classification models.

16.8.7 Countermeasures and Community Directions

Different countermeasures exist in the domain — adversarial retraining, GAN-based defense, hybrid distillation — which can be used to improve the performance of the detection system. Once a defense (for example adversarial retraining) is in place, the bars show that performance increases: when you re-attack the system, both accuracy and robustness improve.

Currently the community is looking into explainability of models — how well you can explain the model — plus incremental learning, labeling, and similar topics. These are the directions the community is moving towards in the next five years.

16.9 Conclusions and Course Wrap-Up

16.9.1 What We Concluded

The conclusions mirror the opening quotes. The 1988 statement that the problem of viruses is temporary and would be solved in two years was absolutely wrong. To win this game, you must first visualize it as a zero-sum game between malware developers and the anti-malware community, and as a good guy building antiviruses you should think, design, and develop one step ahead to stay superior in the game.

The technical conclusions: we can use machine learning, deep learning, data science, and data mining to build effective and efficient malware detection systems — a data-mining-based solution can be proposed to the anti-malware community. But all these models are highly vulnerable to adversarial attacks, so threat modeling is needed to improve both the accuracy and the robustness of the detection system, and doing this threat modeling is an essential step that helps achieve superiority in the detection game.

16.9.2 The TikTok Discussion

The answer doubles as a live example of the course material: spyware and trojans both fit the taxonomy from section 16.2 (behavior without consent, disguise), and the exfiltration to a command and control server is the kind of behavior a dynamic-analysis system logs and a permission model can flag.

16.9.3 Industry Perspective: The ML Component Is Only 20 to 30%

The rule also explains the course's emphasis: the visible 20 to 30% is the model, but the invisible 70 to 80% — data pipelines, retraining on new malware, monitoring, deployment, rollback — decides whether the product actually works. That is why the session walked the full pipeline (data, features, EDA, evaluation) and not only the algorithms.

16.9.4 Thesis Advice: Read 25 Papers First

16.9.5 Course Closing: Learn by Doing

A closing note on learning: whatever strategies were taught in this course — classification, clustering, association rule mining — until you do it yourself and see it yourself, you will not love it. Once you start applying it to solve a real problem, you will start loving it. That is why the course ended with an assignment: the code was roughly 20 to 30 lines only, with 10 days to do it, and the idea was to get you familiar with coding. The concepts stay junk until you code them to solve a real problem.

Real-world: if you take up a data science or data mining based project for your thesis, read papers, and solve a real-life problem, everything studied in this course will be helpful.

Exam Guidance Summary

The exam-related guidance from this session, collected in one place:

• The comprehensive exam covers everything discussed from the first class through this one — modules one through six plus module nine (today's session on data mining applications: malware detection). Whatever was discussed, including today, is part of the comprehensive exam syllabus.
• Modules seven and eight were not discussed in class and are not part of the comprehensive exam syllabus.
• Expect this session's content to be examinable: malware definitions and taxonomy, the zero-sum game framing, the signature/heuristic/behavior engines and their limitations, the five-step data mining pipeline, permission features and the confusion matrix in the malware domain (malware = 1, benign = 0, accuracy formula), why false positive is the most important metric, the Drebin/VirusTotal dataset setup, the results (94% random forest accuracy, AUC 98, 197 to 16 features with less than 1% drop), and the adversarial part (evasion vs poisoning, white/gray/black box, fooling rate, 94% to 50% impact).
• The taxonomy list is worth memorizing: virus attaches to a host, worm runs standalone, Trojan pretends to be useful, spyware spies, ransomware locks and demands money, hybrid combines properties.
• The performance-measure ranking by community convention: minimize the false positive first, then maximize accuracy, then AUC.
• Signature-based detection is the backbone of antiviruses, is human-driven and reactive, and fails against zero-day and quickly evolving malware — expect questions on why.
• No other exam-specific advice was given in this session; the study advice of the course remains: refer to the recorded classes from class one onward.

Key Industry Applications

• Malware detection as a whole is the flagship example: antivirus systems built with data mining, used by real antivirus companies (several antivirus vendors use permission-based detection strategies like the ones discussed here).
• Signature-based engines: hashing of malicious code, human-maintained signature databases — still the backbone of every antivirus, synced to client machines via virus definition updates.
• Google Play Store security: Bouncer scans submitted applications; 1.2 million malicious or unwanted applications were blocked in 2021.
• VirusTotal.com: Google-owned ensemble service of 50 to 60 antiviruses with a public API — used to label benign applications when all engines agree.
• Drebin dataset: benchmark malicious Android dataset with more than 2,000 citations, used across the research literature.
• APKtool: free third-party tool that disassembles Android applications to extract permission usage.
• Reinforcement learning in security: Q-learning, DQN, and actor-critic agents used to automate malware evasion — the same automation is a known adversarial trend in industry.
• The three-Vs framing (volume, velocity, variety) mirrors the big-data challenges of every large-scale security operation.
• Threat modeling terminology (white/gray/black box, evasion vs poisoning, integrity/availability/privacy goals) is used across industry security teams.
• ML Ops and incremental learning: the industry thumb rule that the ML component is only 20 to 30% of an end-to-end pipeline, with the rest — data pipelines, retraining, monitoring — still maturing.