ML for Traffic Classification and Encrypted Traffic Analysis
Prerequisite Knowledge
This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.
Previously Covered in This Subject
- Feature Engineering Pipeline — covered in Lecture 5 (numerical transformations, categorical encoding, n-gram analysis, TF-IDF, dimensionality reduction)
- Supervised vs. Unsupervised Learning Trade-offs — covered in Lecture 6 (algorithm selection criteria, class imbalance handling, SMOTE, ensemble approaches)
- Supervised Learning and Anomaly Detection — covered in Lecture 7 (confusion matrix, precision-recall trade-off, clustering algorithms, Isolation Forest, One-Class SVM)
- Malware Detection and Classification — covered in Lecture 8 (static vs. dynamic analysis, CNN-based malware visualization, LSTM for API sequence analysis)
- Network Intrusion Detection Foundations — covered in Lecture 9 (deep packet inspection, flow-based analysis, protocol analysis)
- ML Approaches for Network Intrusion Detection — covered in Lecture 10 (Random Forest, XGBoost, ensemble learning, feature engineering pipeline, Kitsune framework)
- DNS-Based Threat Detection — covered in Lecture 11 (DGA detection, domain feature engineering, entropy analysis)
- TLS and Encryption Fundamentals — covered in Lecture 12 (TLS protocol evolution, JA3 fingerprinting, CIC Flow Meter feature extraction)
- User Entity Behavior Analytics (UEBA) — covered in Lecture 12 (behavioral biometrics, brute force attack detection)
13.1 Course Context and Remaining Classes
13.1.1 Remaining Schedule
The course is nearing its end with approximately four classes remaining. The schedule going forward is: this class on ML for traffic classification, application identification, and encrypted traffic analysis; followed by two classes on Adversarial Machine Learning Fundamentals; and finally a review class. August 15 is a holiday, and the class for that date will be rescheduled.
13.1.2 Quiz 2 and Mark Distribution
Exam note: A second quiz (Quiz 2) has been added, mandated by the course team. It consists of 20 questions and is worth 5 marks. The quiz window runs from August 10 to August 20. It focuses on situational learning aspects related to the course rather than purely technical content. The overall mark plan is: Quiz 1 = 10 marks, Assignment = 20 marks, Quiz 2 = 5 marks, totaling 35 marks. The pro-rata adjustment for the 5-mark quiz will be decided later.
13.2 Network Traffic Profiling — Recap and Context
13.2.1 Previous Class Recap
The previous class covered user entity behavior analytics (UEBA) and brute force attacks, including a simulation exercise. UEBA focuses on understanding normal versus anomalous behavior patterns of users and entities within a network. The simulation game — a tabletop exercise — modeled incident response, a standard industry practice where IT teams gather to rehearse reactions to security incidents in a controlled setting.
This class builds on that foundation by moving from user behavior to traffic behavior — how to identify what application or threat is running on a network by analyzing the traffic itself. The conceptual shift is important: UEBA asks "is this user acting strangely?", while traffic classification asks "is this network flow carrying something it should not?" Both rely on pattern recognition, but the data source and feature engineering differ completely.
In UEBA, the features are behavioral — login times, file access patterns, privilege usage. In traffic classification, the features are network-level — packet sizes, inter-arrival times, flow durations, and protocol metadata. The underlying machine learning principles (anomaly detection, classification, clustering) remain the same, but the feature space changes entirely.
13.3 Application Identification — Evolution of Approaches
13.3.1 Port-Based Identification
The earliest and simplest method for identifying network applications relied on static port assignments. Standard ports were assigned to protocols during the design of application-layer protocols like HTTP, FTP, SSH, and SMTP. Common well-known ports include:
- Port 22 — SSH
- Port 23 — Telnet
- Port 25 — SMTP
- Port 53 — DNS
- Port 80 — HTTP
- Port 443 — HTTPS
Q: Which port does SSH run on? Which port does HTTPS run on? A: SSH runs on port 22, HTTPS on port 443, HTTP on port 80, SMTP on port 25, DNS on port 53. A common confusion is mixing up DNS (port 53) with Telnet (port 23) — remember: DNS resolves names (53 sounds like "name server"), Telnet provides remote terminal access (23 is the older protocol).
Port-based identification walkthrough. Suppose you are a firewall administrator in the early 2000s and your security policy says "block all file-transfer protocols." With port-based identification, the task is trivial:
- FTP uses ports 20 (data) and 21 (control) — block both.
- SCP/SFTP runs over SSH on port 22 — block port 22 if SSH file transfers are not allowed.
- HTTP uploads on port 80 — block port 80 outbound (or restrict to GET-only via a proxy).
Each application maps to a known port, so identification and enforcement are one-to-one. The firewall rules are simple, deterministic, and fast.
In a port-based approach, identifying the application is trivial: see the port number and you know the application. Blocking an application is equally straightforward — just block the port. If a security policy requires blocking a particular service, a firewall administrator simply closes the corresponding port.
This approach worked well in the early internet era when protocols were well-known and applications respected their assigned ports. However, it became increasingly inadequate as the network landscape evolved.
Scope: Port-based identification assumes applications use their assigned ports. This assumption broke down as applications began using non-standard ports to evade firewalls, and as HTTPS consolidation forced everything onto port 443.
13.3.2 The Consolidation to Port 443
Think of an enterprise network as a mansion with many doors and windows. The "king of the castle" decided to close almost all openings and keep only one main door — HTTPS on port 443 — to simplify inspection and control. Even attackers learned to use the same main door, because it was the only one left open.
Over time, organizations began aggressively blocking non-essential ports as a security measure. ICMP, Telnet, FTP, and even plain HTTP (port 80) became blocked or extinct in many corporate environments. Firewalls enforced rules like the iptables example from the companion material — allowing only specific inbound TCP connections (e.g., SSH from a trusted subnet) and dropping everything else:
# ACCEPT inbound TCP connections from 192.168.100.0/24 to port 22
iptables --append INPUT --protocol tcp --source 192.168.100.0/24
--dport 22 --jump ACCEPT
# DROP all other inbound TCP connections to port 22
iptables --append INPUT --protocol tcp --dport 22 --jump DROPThe consequence was profound: all applications migrated to port 443. File transfers moved to HTTP-based services like OneDrive, Google Drive, and SharePoint. Email shifted to browser-based clients running over HTTPS. Even video streaming, social media, and every other application converged on port 443. The single "main door" carried everything.
Real-world: This port consolidation is why modern enterprise networks can no longer identify applications by port number alone. Port 443 carries legitimate business traffic, personal browsing, file sharing, and malware communications all at once — making port-based identification useless for security enforcement.
Q: Why is YouTube flagged as high risk in application classification? A: Risk is contextual. YouTube is flagged as high risk because it may not be business relevant — employees watching YouTube impacts productivity. Low risk means high business relevance. YouTube is only business-relevant for marketing or promotion teams. The same application can be "low risk" for one department and "high risk" for another. This contextual nature of risk is a recurring theme in network security — classification is never purely technical, it always involves organizational policy.
Attackers exploited this same convergence. Just as legitimate applications used the main door, malware began communicating over HTTPS as well. The result: port 443 carries both good and bad traffic, and there is no way to tell them apart by port number. This is the fundamental problem that drove the need for deeper inspection techniques.
13.3.3 Signature-Based Detection — Open App ID and Snort
The second generation of application identification moved to signature-based detection. Rather than looking at which port the traffic uses, this approach inspects the actual content of the packets — a technique called deep packet inspection (DPI). The idea is straightforward: every application leaves a recognizable "fingerprint" in its packet payloads, just as every person has a distinctive handwriting style.
The Open App ID project, part of the Snort ecosystem, provides an open-source framework for identifying applications through deep packet inspection. The system uses lightweight Lua scripts as detectors — Lua was chosen because packets arrive at high speed (potentially millions per second on a busy link) and must be evaluated at line rate. A heavyweight language would introduce unacceptable latency.
The Open App ID project contains thousands of detectors (over 3,123 at one point) that match packet payloads against known application signatures. These are essentially extended Snort signatures specialized for application identification. Each detector examines packet contents to determine whether the traffic belongs to a specific application — file sharing, video streaming, social media, or other categories.
Real-world: Open App ID is publicly available and allows security teams to write custom Lua-based detectors. The detectors parse packet payloads without encryption and match against known patterns. For example, an HTTP GET request containing a specific User-Agent string or a DNS query to a known domain can be matched against detector rules.
Pitfall: Signature-based detection requires access to the packet payload. If the payload is encrypted (as with TLS/HTTPS), the detector sees only ciphertext — a meaningless string of bytes. This is the Achilles' heel of DPI: as encryption became ubiquitous, signature-based methods became increasingly ineffective.
However, as traffic became encrypted (TLS/HTTPS everywhere), signature-based detectors began failing. You cannot inspect the payload of an encrypted packet. This drove the shift to the third generation: machine learning-based approaches.
13.3.4 The Three-Generation Summary
The evolution of application identification follows three distinct generations:
| Generation | Method | Strengths | Weakness |
|---|---|---|---|
| 1st: Port-based | Static port assignment (port 22 = SSH, port 443 = HTTPS) | Simple, fast, deterministic | Useless once all traffic consolidated to port 443 |
| 2nd: Signature-based | Deep packet inspection using Lua scripts and Snort signatures (Open App ID) | Effective for unencrypted traffic, thousands of known signatures | Fails when payloads are encrypted (TLS/HTTPS) |
| 3rd: ML-based | Statistical and behavioral analysis of flow features | Works on encrypted traffic, learns patterns from data | Requires labeled training data, feature engineering effort |
The transition from each generation was driven by a fundamental limitation of the previous approach. Port-based failed because of port consolidation. Signature-based failed because of encryption. ML-based works because it analyzes flow behavior — packet sizes, timing, sequences — rather than payload content. Each generation addressed the weakness of the previous one.
13.4 Machine Learning for Traffic Classification
13.4.1 Feature Extraction — CICFlowMeter
Feature selection is critical for three reasons: training speed, model interpretability, and avoiding overfitting. CICFlowMeter is an open-source tool that extracts 80+ features from network traffic flows. These features include flow-level statistical parameters (packet counts, byte counts, duration), temporal parameters (inter-arrival times, jitter), and protocol-specific fields.
Real-world: CICFlowMeter is publicly available and widely used in network traffic research. It extracts features from pcap files or live network traffic, providing a ready-made feature set for ML-based traffic classification.
Q: Should I just use all 80+ features from CICFlowMeter directly? A: Do not use them blindly. Understand your use case, think through which features are relevant, and then select accordingly. Feature selection is critical — it impacts training speed, interpretability, and overfitting. Using all features adds noise, slows training, and makes the model harder to interpret. Select features that are relevant to your specific classification task.
The companion material on network traffic profiling emphasizes that feature engineering is one of the most important steps in any ML pipeline. Tools like CICFlowMeter extract features such as:
- Flow-level: Total packets, total bytes, flow duration
- Temporal: Inter-arrival time statistics (mean, std, min, max), jitter
- Protocol-specific: TCP window size, flag counts (SYN, ACK, FIN, RST)
- Statistical: Packet length statistics, payload size distributions
The key is to match features to the problem. For video conferencing vs. file transfer classification, temporal features (jitter, inter-arrival times) matter more than raw byte counts. For detecting C2 beaconing, periodicity in inter-arrival times is the critical signal.
13.4.2 Random Forest for Traffic Classification
Random Forest is highlighted as one of the best-performing algorithms for traffic classification, achieving 95 to 100% accuracy in general use cases. A Random Forest is an ensemble of decision trees — each tree votes on the classification, and the majority vote wins. The "random" part comes from two sources of randomness: each tree is trained on a random subset of the data (bootstrap sampling), and at each split, only a random subset of features is considered.
Why Random Forest works well for traffic classification:
- Handles high dimensionality: Effectively manages 100+ features without overfitting, because each tree only sees a subset of features at each split
- Feature importance rankings: You can determine which protocol features matter most for classification — this is invaluable for understanding what distinguishes, say, video conferencing from file transfer traffic
- Parallel training: Each tree is independent, so training scales well across multiple cores
- Real-time inference: Classification of a single flow takes under millisecond — fast enough for production deployment
Feature importance example: If your goal is to identify video conferencing protocols (Teams, Zoom) versus file transfer protocols, Random Forest can rank which features distinguish them. Video conferencing traffic is characterized by regular, small packets with low jitter (real-time media), while file transfer traffic shows larger packets with bursty patterns. The feature importance ranking tells you which features the model relies on most — if "inter-arrival time standard deviation" ranks high, that confirms the temporal distinction between the two traffic types.
Q: What are overfitting and underfitting? A: Overfitting means the model memorizes the training data patterns so thoroughly that it works superbly on training data but fails on unseen test data and real-world scenarios. The model becomes too specific to the traffic it has seen and completely misses new patterns. Think of it like a student who memorizes exam answers without understanding the concepts — they ace the practice test but fail the real exam when the questions change slightly.
Underfitting is the opposite — the model is too simple to capture the underlying patterns at all. It performs poorly on both training and test data. This is like using a straight line to fit data that follows a curve — the line misses the pattern entirely.
Both must be avoided. The goal is a model that generalizes — one that captures the true patterns in traffic behavior, not the noise in the training data.
Pitfall: Overfitting is particularly dangerous in traffic classification because network traffic patterns change over time. A model that overfits to today's traffic may fail completely when new applications or attack techniques emerge. Always evaluate on a held-out test set and consider temporal splits (train on older data, test on newer data) to assess generalization.
13.4.3 XGBoost for Traffic Classification
XGBoost (Extreme Gradient Boosting) is another effective algorithm for traffic classification with specific advantages. While Random Forest builds trees independently and averages them, XGBoost builds trees sequentially — each new tree corrects the errors of the previous ones. This boosting strategy often achieves higher accuracy than a single Random Forest.
XGBoost advantages for traffic classification:
- Handles extreme class imbalance: Network traffic datasets often have very unequal representation across application classes — HTTP might dominate 90% of traffic while DNS is 1% and malware is 0.01%. XGBoost assigns higher weights to minority-class samples, preventing the model from ignoring rare but important classes
- Built-in handling of missing values: In practice, certain parts of traffic data may be incomplete or difficult to simulate — XGBoost learns optimal default directions for missing values during training
- Built-in regularization: L1 and L2 regularization terms in the objective function prevent overfitting, which is critical when working with high-dimensional feature spaces
The combination of class imbalance handling and regularization makes XGBoost particularly well-suited for real-world traffic datasets where rare applications or threats are underrepresented.
13.4.4 Other ML Algorithms
Support Vector Machines (SVM) and K-Nearest Neighbors (KNN) also apply to traffic classification but with different trade-offs.
SVM works by finding the optimal hyperplane that separates classes with the maximum margin. In traffic classification, SVMs are effective when the feature space is well-structured and classes are separable. However, SVMs can be slow to train on very large datasets and require careful kernel selection.
KNN classifies a new flow by finding the K most similar flows in the training data and taking a majority vote. KNN is slower at inference time because it must compute distances to all training samples, but it is well-suited for dynamic systems where traffic patterns evolve over time — new patterns are automatically incorporated without retraining. The companion material on network traffic profiling notes that k-means and related clustering approaches are widely used for grouping similar network connections, though they require careful selection of the parameter K.
Pitfall: KNN's inference time scales linearly with the size of the training set. For a dataset with millions of flows, classifying a single new flow requires computing distances to all of them. This makes KNN impractical for real-time classification on high-volume links without approximation techniques (e.g., KD-trees or locality-sensitive hashing).
13.4.5 Deep Learning Architectures
Deep learning approaches, particularly CNNs and LSTMs, are applied to traffic classification when spatial patterns and temporal dependencies matter.
CNN for traffic: Convolutional neural networks can extract spatial features from traffic data, similar to how they process images. In earlier lectures on malware detection, CNNs were applied to malware file visualizations — analyzing how pixels are organized to detect anomalies. For network traffic, CNNs can capture spatial patterns in packet sequences. The idea is to treat a sequence of packets as a one-dimensional "image" and apply convolutional filters to detect local patterns — for example, a pattern of small-large-small packet sizes might indicate a specific application protocol.
LSTM for traffic: Long Short-Term Memory networks excel at capturing temporal dependencies — what happens before and what happens next. In network traffic, sequence matters critically: a login event followed by file export is normal; file export followed by login is suspicious. LSTMs process sequential streams and capture these temporal relationships.
Spatial vs. temporal patterns in security:
- Spatial patterns (CNN): How bytes or packets are organized in a single observation — malware file analysis is primarily spatial
- Temporal patterns (LSTM): How events unfold over time — network traffic analysis is primarily temporal
- The choice between CNN and LSTM depends on which type of pattern dominates your problem
Exam note: LSTM-based approaches for threat detection (such as ransomware) achieve to accuracy. This is because ransomware produces a distinctive temporal signature — a burst of file access operations followed by sequential encryption — that LSTMs can learn to recognize.
13.4.6 SDN Integration
Software-Defined Networking (SDN) provides a deployment context for ML-based traffic classifiers. In SDN architectures, the control plane (which decides how to handle traffic) is separated from the data plane (which forwards traffic). This separation allows centralized traffic management and dynamic policy enforcement.
How ML classifiers integrate with SDN:
- The SDN controller receives flow statistics from switches
- An ML classifier analyzes the flow features in real time
- If the classifier identifies suspicious traffic, it instructs the SDN controller to take action (block, quarantine, redirect)
- This enables automated, real-time security responses without manual intervention
This is a use-case consideration rather than a core ML technique — the ML algorithms themselves are the same (Random Forest, XGBoost, LSTM), but SDN provides the infrastructure to deploy them at scale with automated enforcement.
13.5 Encrypted Traffic Analysis — Side-Channel Features
Since encryption prevents payload inspection, encrypted traffic analysis relies on side-channel features — observable properties of the traffic that do not require decryption. The key insight is: even without seeing the content, the behavior of the traffic reveals the application. This is the fundamental premise that makes the entire field of encrypted traffic analysis possible.
13.5.1 Packet Size and Length Sequences
Different applications produce characteristic packet size patterns because the underlying data they transmit has different structure. A short text tweet generates small packets. A video stream generates medium-sized, regular packets. A file upload generates large, sustained bursts. These patterns are visible even when the payload is encrypted, because the packet length reflects the original data size.
Packet size fingerprinting — Twitter vs. Pastebin vs. malware. Consider three users on the same corporate network, all using HTTPS on port 443:
Twitter/social media: Posts are small — short text messages produce small packets (typically 200–500 bytes). The packet size distribution reflects small, frequent uploads interspersed with small responses (timeline refreshes). The pattern is: many small packets in both directions.
Pastebin: Pastebin is a tool where users paste any content and share a link. It became extremely popular, even among attackers, for exfiltrating code, images, or data. Pastebin uploads are larger than tweets — a code paste might be 5–50 KB, producing medium-to-large packets in the upload direction, followed by a small confirmation response. The pattern is: fewer but larger upload packets compared to Twitter.
Malware (C2 communication): Malware communicating with its command-and-control server typically sends small, periodic keep-alive packets (100–300 bytes) at regular intervals, with occasional larger packets when receiving commands or exfiltrating data. The pattern is: small, regular, metronomic.
Without decrypting the content, without seeing what is inside the packets, you can derive application behavior purely from packet size and sequence patterns. A classifier trained on these features can distinguish Twitter from Pastebin from malware with high accuracy.
The companion material on network traffic analysis confirms this approach: even when TLS encrypts the payload, features like packet size, direction, and timing remain observable. The key is that encryption hides the content but not the behavior.
13.5.2 Inter-Arrival Times and C2 Beaconing
The timing between packets — inter-arrival times — reveals application behavior. Different applications have fundamentally different timing patterns because of how they interact with users and servers.
C2 beaconing detection: A critical application of inter-arrival time analysis is detecting Command and Control (C2) beaconing. If inter-arrival times show periodicity — a cyclic, regular pattern — this strongly suggests C2 beaconing. Malware infected on a machine periodically "calls home" to its C2 server at regular intervals (e.g., every 30 seconds, every 5 minutes). This periodic heartbeat is a signature that legitimate applications rarely produce.
Why legitimate traffic is not periodic: Legitimate applications have bursty, irregular traffic patterns. A user clicks a link — traffic spikes. They read the page — traffic goes quiet. They scroll — more traffic. This burst-and-pause pattern reflects human interaction. C2 beaconing, by contrast, produces metronomic, clock-like traffic — the malware does not wait for a human, it runs on a timer.
Detecting beaconing from inter-arrival times. Suppose you observe the following inter-arrival times (in seconds) for a connection:
- Connection A: 0.3, 12.5, 0.1, 45.2, 0.2, 8.7, 0.1 — highly irregular, bursty → legitimate user traffic
- Connection B: 30.0, 30.1, 29.9, 30.0, 30.1, 30.0 — nearly constant, periodic → likely C2 beaconing
The standard deviation of inter-arrival times is a simple feature that captures this distinction. Connection A has high variance (bursty); Connection B has near-zero variance (periodic). More sophisticated approaches use Fourier analysis or autocorrelation to detect periodicity even when the beaconing interval drifts slightly.
Real-world: Detecting this periodicity is a powerful indicator of compromise without ever decrypting the traffic. Security tools compute inter-arrival time statistics and flag connections with suspiciously low variance or strong periodic components.
13.5.3 MTU-Based Application Fingerprinting
Maximum Transfer Unit (MTU) patterns from packet protocol headers provide another fingerprinting dimension. The MTU defines the largest packet size a network link can carry — typically 1500 bytes for Ethernet. Applications that produce data larger than the MTU must fragment it into multiple packets, and the way they fragment reveals the application's behavior.
MTU fingerprinting — YouTube vs. malware vs. Dropbox. Consider three encrypted connections on the same network:
- YouTube (video streaming): Produces regular-sized chunks — consistent streaming data packets reflecting video buffering behavior. YouTube's adaptive bitrate streaming sends data in predictable chunks that match the video encoding rate. Packet sizes cluster around 1400–1500 bytes (near MTU), arriving at regular intervals.
- Malware (C2 beaconing): Produces smaller, irregular chunks characteristic of beaconing activity — small keep-alive packets (100–300 bytes) sent to the C2 server. These are well below MTU because the beaconing message is tiny.
- Dropbox (file upload): Produces patterns consistent with file uploads — large, sustained bursts of MTU-sized packets as the file is broken into maximum-sized segments and sent as fast as the network allows. The pattern is: a burst of 1400–1500 byte packets, then a pause, then another burst.
The distinction between these three patterns is visible without decryption. A simple classifier using packet size distribution and timing features can distinguish streaming from beaconing from file transfer with high accuracy.
Exam note: This distinction — regular streaming chunks versus small beaconing chunks — is a key concept for encrypted traffic analysis. The behavior is difficult for malware to mimic because the underlying communication pattern is fundamentally different. Video streaming must send large, regular chunks to maintain playback quality; beaconing must send small, periodic messages to maintain C2 contact. These constraints make the patterns reliable classifiers.
13.6 TLS Handshake Metadata for Fingerprinting
13.6.1 The TLS Handshake Process
The TLS handshake is the negotiation that occurs before encrypted communication begins. Because the handshake itself is not encrypted (encryption has not started yet), its metadata is fully visible and extremely valuable for fingerprinting.
Critical point: The encryption itself has not started yet during the TLS handshake. The handshake metadata is visible because the negotiation happens before encryption begins. This is the foundation of encrypted traffic fingerprinting — the "negotiation about how to encrypt" is itself unencrypted.
The process works as follows:
- The client sends a SYN packet to initiate the TCP connection.
- The server responds with SYN-ACK (TCP handshake).
- The client sends a Client Hello message — this contains the TLS version, a list of cipher suites the client supports, extensions, elliptic curves, and point formats.
- The server responds with a Server Hello — this contains the server's chosen cipher suite, its certificate, and a list of supported cipher suites.
- The client selects cipher suites it can support (e.g., elliptic curve, RSA) and sends its certificate if required.
- Both parties agree on an encryption algorithm and begin encrypted communication.
Think of it like two people who do not share a common language. They wave hands, then one opens a paper listing all the languages they speak. The other responds with their list. They find the common language and begin conversing. The TLS handshake works identically — the metadata exchanged before encryption starts is the "language negotiation." Just as you can guess a person's nationality from the list of languages they speak, you can guess the application (or malware family) from the list of cipher suites it supports.
13.6.2 Fingerprinting Legitimate vs. Malicious Traffic
The key premise: both legitimate websites (Facebook, YouTube, internal applications) and malware use TLS. Both go through the TLS handshake. But the metadata they expose differs — and this difference is what makes fingerprinting possible.
What differs between legitimate and malicious TLS metadata:
- Legitimate websites use well-known certificates issued by trusted Certificate Authorities (CAs), modern cipher suites (TLS 1.3, AES-256-GCM), and standard TLS configurations that match popular browsers and servers
- Malware may use self-signed certificates (no CA trust chain), outdated cipher suites (TLS 1.0, RC4), unusual TLS configurations, or custom implementations that produce distinctive handshake patterns
By fingerprinting the TLS handshake metadata of known-good applications, you create a baseline. When traffic deviates from this baseline — different certificates, unusual cipher choices, unexpected TLS configurations — you flag it as potentially malicious.
Real-world: This approach works without ever decrypting the traffic content. You are analyzing the metadata of the negotiation itself, not the encrypted payload. The companion material on network security confirms that TLS encapsulation prevents packet sniffers from obtaining useful information from the encrypted content, but the handshake metadata — visible before encryption starts — provides a rich fingerprinting surface.
13.6.3 TLS Fingerprinting Tools
Several open-source tools implement TLS fingerprinting:
Joy: An early open-source tool for capturing and analyzing TLS metadata. It takes captured packets, performs sequence analysis on inter-arrival IP packets, and builds a TLS fingerprint database. The database is stored as a gzip file that can be unpacked and examined. Joy was one of the first tools to systematically fingerprint TLS traffic and remains useful for research.
JA3 and JA3+: Methods for creating TLS fingerprints based on the Client Hello message. JA3 extracts five fields from the Client Hello:
- TLS version
- Cipher suites (ordered list)
- Extensions
- Elliptic curves
- Point formats
These five fields are concatenated and hashed (MD5) to create a 32-character fingerprint that uniquely identifies a TLS client. For example, Chrome on Windows produces a different JA3 hash than Firefox on Linux, because they implement TLS differently — different cipher suite preferences, different extension orders, different supported curves.
JA3 fingerprinting in practice. When a new TLS connection is observed, the security tool extracts the Client Hello fields, computes the JA3 hash, and compares it against a database of known fingerprints:
- Known Chrome 120 on Windows:
e7d705a3286e19ea42f587b344ee6865 - Known Firefox 121 on Linux:
b32309a26951912be7dba376398abc3b - Unknown application:
a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6
If the unknown fingerprint does not match any known browser or application, it is flagged for investigation. This is how security teams identify malware using custom TLS implementations.
Real-world: JA3 fingerprinting is widely used in production security tools. The JA3 fingerprint database contains known fingerprints for browsers, malware families, and applications. When new traffic is observed, its JA3 fingerprint is compared against the database for identification.
Zeek: A more recent, general-purpose network analysis framework that includes TLS fingerprinting capabilities among many other features. Zeek (formerly Bro) is deployed at network monitoring points and generates rich logs of network activity, including TLS metadata.
Mercury: A specialized tool for encrypted traffic analysis and TLS fingerprinting, representing more modern alternatives to Joy. Mercury focuses specifically on extracting features from encrypted traffic for classification.
13.6.4 Practical Fingerprinting Demonstration
The lecture demonstrated TLS fingerprinting using browser fingerprints as examples. Chrome, Safari, and other browsers produce distinct TLS fingerprints because they implement TLS differently — different cipher suite preferences, different extension orders, different supported curves. These differences, invisible to the user, are machine-readable and distinguishable.
Browser fingerprint comparison. Consider the Client Hello from two browsers connecting to the same website:
- Chrome 120: Supports cipher suites in this order: TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256. Extensions include: server_name, extended_master_secret, renegotiation_info, supported_groups (x25519, secp256r1, secp384r1).
- Safari 17: Supports cipher suites in a different order: TLS_AES_128_GCM_SHA256, TLS_CHACHA20_POLY1305_SHA256, TLS_AES_256_GCM_SHA384. Extensions include: server_name, extended_master_secret, renegotiation_info, supported_groups (x25519, secp256r1, secp384r1, secp521r1).
The cipher suite order differs. The supported groups differ slightly. These differences produce different JA3 hashes, allowing the classifier to distinguish Chrome from Safari — even though both are legitimate browsers.
The fingerprinting database grows as more applications are catalogued. Each new application's TLS metadata is captured, processed, and added to the reference database. Observed traffic is then matched against this database for identification. This is an ongoing process — as new versions of browsers and applications are released, their fingerprints must be updated in the database.
13.7 Encrypted Threat Scenarios
13.7.1 Ransomware Detection
Ransomware produces distinctive observable traffic patterns even when encrypted. The encryption of the ransomware's command-and-control traffic does not hide the behavioral signature of the attack — the pattern of file operations is visible in the network flow.
Ransomware traffic signature:
- File access burst: A sudden burst of file access operations as the ransomware scans the file system to enumerate targets. This manifests as a spike in SMB (Server Message Block) or NFS traffic — hundreds or thousands of file metadata queries in rapid succession.
- File encryption requests: Repeated sequential file operations consistent with encrypting files one by one. The ransomware reads each file, encrypts it, and writes the encrypted version back — producing a read-write-read-write pattern that is distinct from normal application behavior.
Ransomware detection from traffic patterns. Consider a workstation that suddenly begins the following network activity:
- 14:00:00 — Burst of 500 SMB file enumeration requests (scanning the file share)
- 14:00:05 — Sequential read-write pairs on 200 files (encrypting documents)
- 14:00:30 — A single outbound HTTPS connection to an unknown IP (C2 check-in)
- 14:00:35 — A ransom note file appears on the file share
The burst of file enumeration followed by sequential read-write pairs is the distinctive signature. A human analyst can recognize this pattern, but an LSTM can learn it automatically from training data and detect it in real time — even before the ransom note appears.
Exam note: A question from the midterm exam asked: "If you are seeing this script function often, what could it be?" — pointing to ransomware. The answer involves recognizing the pattern of repeated file encryption operations. LSTM-based temporal analysis achieves to accuracy for ransomware detection by capturing the sequential pattern of file operations over time. The LSTM learns that the sequence "enumerate → read → write → read → write → ..." is characteristic of ransomware, while "read → think → read → write → read → ..." is characteristic of normal user behavior.
13.7.2 Crypto Mining Detection
Q: What is crypto mining? A: Crypto mining is the process of using computational resources to solve cryptographic puzzles for cryptocurrency rewards. The mining process requires enormous computational power — miners compete to find a hash value below a target threshold. The first to find it earns the cryptocurrency reward (e.g., Bitcoin, Ethereum). Attackers exploit this by finding exposed virtual machines on the internet, installing mining software, and mining at the victim's expense — consuming the victim's CPU/GPU resources and electricity.
Crypto mining is the process of using computational resources to solve cryptographic puzzles for cryptocurrency rewards. Attackers seek exposed virtual machines on the internet, install bitcoin mining software, and mine cryptocurrency at the victim's expense — consuming the victim's CPU/GPU resources and electricity.
Crypto mining traffic signature: The observable traffic signature of crypto mining is persistent pool connections — the mining software maintains a continuous, long-lived connection to a mining pool server. Unlike normal user traffic, which is bursty and intermittent, crypto mining traffic is a constant, steady stream. The miner must continuously receive new work units from the pool and submit solved hashes — this produces a steady bidirectional flow with minimal idle time.
Real-world case study: Cloud VM auto-scaling attack. A customer experienced an unusual situation where their cloud virtual machine was spinning up new instances automatically. They had not configured auto-scaling for heavy traffic, yet new instances kept appearing. The investigation unfolded as follows:
- Observation: The cloud bill tripled in one month despite no change in application traffic.
- Investigation: Network monitoring revealed persistent outbound connections from multiple VMs to a known mining pool IP address (e.g.,
pool.minexmr.com:4444). - Root cause: Crypto mining malware had infected the original VM through an exposed management port. The malware consumed 100% CPU, triggering the auto-scaling system to spawn additional VMs to handle the perceived load. The malware then spread to the new VMs via the same vulnerability.
- Traffic indicator: The persistent pool connections — long-lived, steady, to a known mining pool — were the traffic signature that helped identify the compromise. Normal application traffic is bursty and targets diverse endpoints; mining traffic is persistent and targets a single pool server.
This case illustrates how crypto mining creates a double impact: direct resource theft (CPU cycles for mining) and indirect cost amplification (auto-scaling spinning up additional instances that also get infected). The total cost was not just the stolen compute — it was the multiplied cloud bill from auto-scaling.
Pitfall: Do not confuse persistent pool connections with legitimate long-lived connections. Streaming services (Netflix, YouTube) also maintain long-lived connections, but they target content delivery networks (CDNs), not mining pools. The combination of long-lived connection + known mining pool destination + constant CPU usage is the reliable signature. Any single feature alone is insufficient.
13.8 Self-Supervised Learning (Introduction)
13.8.1 Emerging Approach for Traffic Analysis
Self-supervised learning is mentioned as an emerging approach for encrypted traffic analysis. While not covered in depth in this lecture, it represents the next evolution in the field — models that can learn traffic representations from unlabeled data, reducing the need for manually labelled training datasets.
Why self-supervised learning matters for traffic classification: The biggest bottleneck in ML-based traffic classification is labeled data. To train a supervised model (Random Forest, XGBoost, LSTM), you need thousands of labeled examples — "this flow is YouTube," "this flow is malware." Labeling network traffic is expensive, time-consuming, and requires domain expertise. Self-supervised learning sidesteps this by learning useful representations from raw, unlabeled traffic data. The model learns what "normal" traffic looks like without any labels, and then fine-tuning on a small labeled dataset achieves good classification performance.
This approach is particularly relevant for encrypted traffic analysis because:
- Label scarcity: Labeled traffic datasets are rare and expensive to create. Capturing and labeling malware traffic requires controlled environments and expert analysis.
- Concept drift: Traffic patterns change over time as applications update, new applications emerge, and attackers evolve their techniques. Models trained on old labeled data degrade. Self-supervised models can continuously learn from new unlabeled data.
- Zero-day detection: Self-supervised models learn the structure of normal traffic, making them potentially better at detecting novel anomalies that supervised models would miss (since supervised models only learn to distinguish classes present in the training data).
This will be explored in subsequent classes.
13.9 Feature Categories for Encrypted Traffic Analysis
13.9.1 Feature-to-Intelligence Mapping
The mapping of feature categories to observables and derived intelligence provides a comprehensive view of what can be learned from encrypted traffic without decryption:
| Feature Category | Observable | Derived Intelligence |
|---|---|---|
| Packet size | Length of each packet | Application type (social media, file transfer, streaming) — small packets suggest text/chat; large packets suggest file transfer or streaming |
| Packet sequence | Order and pattern of packets | Application behavior fingerprint — request-response patterns, upload/download ratios |
| Inter-arrival time | Time between consecutive packets | C2 beaconing detection (periodic patterns), real-time vs. batch communication |
| TLS handshake metadata | Cipher suites, certificates, extensions | Legitimate vs. malicious application fingerprint via JA3/JA3+ hashes |
| Flow duration | Total connection duration | Long-lived connections suggest persistent activity (crypto mining, C2) or streaming |
| Connection destinations | IP addresses and geolocation | Threat mapping and known-bad infrastructure identification — connections to known mining pools, bulletproof hosting, or Tor exit nodes |
No single feature is sufficient. These features work together — their combination provides robust classification even without decrypting the traffic content. For example:
- A long-lived connection to a known mining pool IP + steady inter-arrival times + constant CPU usage → crypto mining
- Small periodic packets + unknown TLS fingerprint + self-signed certificate → likely C2 beaconing
- Large bursty packets + known CDN destination + irregular timing → legitimate streaming
Each feature alone has too many false positives. The combination is what makes classification reliable.
Combining features for classification. Consider two encrypted connections, both on port 443:
Connection A:
- Packet sizes: 1400–1500 bytes (near MTU)
- Inter-arrival times: 30–50 ms, irregular
- Flow duration: 45 minutes
- Destination: Known CDN IP (Akamai)
- TLS fingerprint: Matches Chrome 120
Connection B:
- Packet sizes: 100–300 bytes
- Inter-arrival times: 30.0 ± 0.1 seconds, periodic
- Flow duration: 24 hours (persistent)
- Destination: Unknown IP in a bulletproof hosting range
- TLS fingerprint: Does not match any known browser
Connection A is likely a video stream (large, irregular packets to a CDN with a known browser fingerprint). Connection B is likely C2 beaconing (small, periodic packets to an unknown host with an unrecognized TLS fingerprint). No single feature makes this distinction — it is the combination that enables accurate classification.
Recap: The six feature categories — packet size, packet sequence, inter-arrival time, TLS handshake metadata, flow duration, and connection destinations — form the foundation of encrypted traffic analysis. Together they provide enough information to classify applications and detect threats without ever seeing the encrypted content. This is why ML-based traffic classification works even in an era of ubiquitous encryption.
Exam Guidance Summary
- Quiz 2: 20 questions, 5 marks, window August 10–20, focuses on situational learning aspects. Mandatory to take.
- Mark distribution: Quiz 1 (10) + Assignment (20) + Quiz 2 (5) = 35 marks total. Pro-rata adjustment for Quiz 2 to be decided.
- Ransomware detection patterns: Expect questions on recognizing file access bursts and repeated encryption requests as indicators of ransomware. LSTM achieves 95–97% accuracy.
- Encrypted traffic analysis: TLS handshake metadata fingerprinting is a key concept — understand that the handshake metadata is visible before encryption begins. JA3 extracts five fields from the Client Hello.
- Application identification evolution: Know the three generations (port-based → signature-based → ML-based) and why each transition was necessary (port consolidation, encryption).
- Feature selection: Understand why blindly using all features is problematic — impacts training speed, interpretability, and overfitting.
- Side-channel features: Packet sizes, inter-arrival times, and TLS metadata are the primary side-channel features for encrypted traffic analysis. No single feature is sufficient — combinations are needed.
- C2 beaconing: Periodic inter-arrival times with low variance are a strong indicator of compromise. Legitimate traffic is bursty and irregular.
- Crypto mining: Persistent pool connections to known mining pool servers, combined with constant CPU usage and auto-scaling exploitation.
Key Industry Applications
- CICFlowMeter — Open-source tool extracting 80+ flow features from network traffic, widely used in research and production. Extracts features from pcap files or live traffic.
- Open App ID / Snort — Signature-based application detection using Lua scripts, part of the Snort IDS ecosystem. Over 3,123 detectors for application identification.
- Joy — Early open-source TLS fingerprinting tool for capturing and analyzing TLS metadata. Builds fingerprint databases from captured packets.
- JA3 / JA3+ — TLS fingerprinting methods creating unique hashes from Client Hello fields (version, cipher suites, extensions, curves, point formats), widely deployed in production security tools.
- Zeek — Modern network analysis framework with TLS fingerprinting and broader traffic analysis capabilities. Generates rich logs of network activity.
- Mercury — Specialized encrypted traffic analysis tool, more recent alternative to Joy.
- Pastebin — Example of a legitimate tool repurposed by attackers for data exfiltration, illustrating the contextual nature of application risk.
- SDN (Software-Defined Networking) — Deployment context for ML-based traffic classifiers, enabling centralized traffic management and dynamic policy enforcement.
- Crypto mining malware — Real-world threat exploiting exposed cloud VMs, with persistent pool connections as the traffic signature and auto-scaling cost amplification as a secondary impact.
AMTCS Lecture 13 notes · ML for Traffic Classification and Encrypted Traffic Analysis
Sections Breakdown
Course schedule overview and Quiz 2 mark distribution
Recap of UEBA and transition to traffic behavior analysis
Three generations: port-based, signature-based, and ML-based identification
Random Forest, XGBoost, SVM, KNN, CNN, and LSTM for traffic classification
Packet size, inter-arrival times, and MTU-based fingerprinting
JA3/JA3+ fingerprinting, Joy, Zeek, and Mercury tools
Ransomware and crypto mining detection from traffic patterns
Emerging approach for traffic analysis with unlabeled data
Six feature categories and their combinations for classification
Exam Revision Notes
Below is the distilled, exam-ready core. Every entry comes from the full explanation above. Use this section for rapid review; return to the main notes when a point needs more context.
Network Traffic Profiling — Recap and Context
Must-know: The conceptual shift from UEBA (user behavior) to traffic classification (network flow behavior) — same ML principles, different feature space.
Self-check: What is the key difference between UEBA and traffic classification in terms of data source?
Connects to: 13.3
Application Identification — Evolution of Approaches
Must-know: Three generations of application identification: port-based → signature-based → ML-based. Each transition was driven by a fundamental limitation (port consolidation, encryption).
Top pitfall: Confusing DNS (port 53) with Telnet (port 23). Assuming port-based identification still works in modern networks.
Self-check: Why did signature-based detection (Open App ID) become ineffective, and what replaced it?
Connects to: 13.4, 13.5
Machine Learning for Traffic Classification
Must-know: Random Forest achieves 95-100% accuracy, handles 100+ features, provides feature importance. XGBoost handles class imbalance and missing values. LSTM achieves 95-97% for ransomware detection. Feature selection is critical — do not use all CICFlowMeter features blindly.
Top pitfall: Using all 80+ CICFlowMeter features without selection (adds noise, slows training). Overfitting to training data without temporal validation.
Self-check: Why is Random Forest well-suited for traffic classification with 100+ features?
Connects to: 13.5, 13.7
Encrypted Traffic Analysis — Side-Channel Features
Must-know: Side-channel features (packet size, inter-arrival times, MTU patterns) reveal application behavior without decryption. C2 beaconing shows periodic inter-arrival times (low variance). YouTube = regular large chunks; malware = small periodic packets; Dropbox = large bursty uploads.
Top pitfall: Assuming encryption hides all traffic characteristics — it hides content but not behavior (size, timing, sequence).
Self-check: How can you detect C2 beaconing from encrypted traffic without decrypting it?
Connects to: 13.6, 13.7
TLS Handshake Metadata for Fingerprinting
Must-know: TLS handshake metadata is visible before encryption begins. JA3 extracts 5 fields from Client Hello (version, cipher suites, extensions, curves, point formats) and hashes them to create a unique fingerprint. Legitimate traffic matches known browser fingerprints; unknown fingerprints are flagged.
Top pitfall: Assuming TLS encryption hides all metadata — the handshake itself is unencrypted. Confusing JA3 with JA3+ (JA3+ adds additional fields).
Self-check: What five fields does JA3 extract from the TLS Client Hello to create a fingerprint?
Connects to: 13.5, 13.7
Encrypted Threat Scenarios
Must-know: Ransomware: file access burst + sequential read-write pairs. LSTM achieves 95-97% accuracy. Crypto mining: persistent pool connections (long-lived, steady, to known mining pool). Auto-scaling attack: mining malware triggers cloud auto-scaling, multiplying costs.
Top pitfall: Confusing persistent pool connections with legitimate long-lived streaming connections. Single features are insufficient — need combination of connection duration + destination + CPU usage.
Self-check: What are the two traffic indicators of ransomware, and how does crypto mining create a 'double impact'?
Connects to: 13.5, 13.6
Self-Supervised Learning (Introduction)
Must-know: Self-supervised learning learns traffic representations from unlabeled data, addressing label scarcity, concept drift, and zero-day detection challenges.
Self-check: Why is self-supervised learning particularly relevant for encrypted traffic analysis?
Connects to: 13.4
Feature Categories for Encrypted Traffic Analysis
Must-know: Six feature categories: packet size, packet sequence, inter-arrival time, TLS handshake metadata, flow duration, connection destinations. No single feature is sufficient — combinations are needed for reliable classification.
Top pitfall: Relying on a single feature for classification — too many false positives. Need feature combinations.
Self-check: Name the six feature categories for encrypted traffic analysis and explain why no single feature is sufficient.
Connects to: 13.5, 13.6
Was this lecture useful?
BitsNotes AI Assistant
Subject Notes AssistantConfigure AI Chat
Choose how to access the chatbotSigned in as
Powered by BitsNotes — 20 messages per day. No API key needed. Want unlimited access? Use "Bring Your Own Key" mode.
Sign in to use AI Chat
Get 20 free AI messages per day to ask questions about your lecture notes. Sign in with Google or GitHub — it takes 5 seconds.
Sign In to BitsNotesSwitch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.