Network Traffic Profiling and Encrypted Traffic Analysis
Network Traffic Profiling and Encrypted Traffic Analysis
12.1 Network Traffic Profiling — Motivation and Concept
12.1.1 Why Traffic Has Patterns
Hook: Why do security teams need to study "normal" traffic before they can spot "abnormal" traffic? Because every network has a rhythm — and if you do not know the rhythm, you cannot hear the wrong note.
Network traffic profiling is the practice of studying the inherent patterns in network traffic to understand what is normal for a given environment. Every organization's network has characteristic rhythms shaped by its business type, workforce, and applications. Just as city roads have predictable congestion patterns — tech parks choke during office commute hours, schools follow morning-afternoon cycles, malls spike on weekends, and food courts peak at lunchtime — an organization's network traffic exhibits patterns tied to its work. A company where salespeople are on the road checking a mobile app will produce different traffic than a company where most employees have desk jobs pushing documents and code around. Fully remote organizations generate yet another pattern. Understanding these patterns is the foundation of traffic profiling.
Intuition + Analogy: Think of network traffic like the water supply in a large apartment building. Each apartment has a typical daily usage pattern — showers in the morning, cooking in the evening, low flow at night. If one apartment suddenly starts running water 24/7 at ten times the normal rate, the building manager notices. The manager does not need to inspect what is being washed — the pattern of flow itself is the signal. Network traffic profiling works the same way: you measure the flow (packet sizes, timing, direction, protocols) and learn what is typical. A deviation from that typical pattern — a sudden spike in connections, an unusual destination, traffic at odd hours — becomes the anomaly you investigate.
The goal is to establish a scientific or technical method for characterizing traffic so that deviations can be detected. Profiling in general English means studying an entity's characteristics and patterns so you can manage it better. Applied to network traffic, profiling means encoding the observable characteristics of traffic flows — size, timing, direction, protocol metadata — into structured representations that tools and models can reason about. This is essential groundwork for the next class's topic: machine-learning-based encrypted traffic classification.
Formalize: A traffic profile is a statistical summary of a set of network flows over a defined time window. The key observable features include:
- Flow duration — how long a connection lasts
- Packet count — number of packets in each direction
- Byte volume — total bytes transferred per direction
- Inter-arrival times — the time gaps between successive packets
- Protocol metadata — TCP flags, port numbers, TLS version, cipher suites
- Directionality — ratio of inbound to outbound traffic
These features are aggregated (often as distributions: mean, variance, percentiles) over a time window (e.g., 5 minutes, 1 hour) to form a profile for that window. The profile becomes the baseline against which future windows are compared.
Worked Example: Consider a small office with 50 employees. Over a typical workday (9 AM–6 PM), the network profile might look like:
| Feature | Typical Value |
|---|---|
| Total outbound bytes (hourly) | 2–5 GB |
| Total inbound bytes (hourly) | 10–30 GB |
| Top destination ports | 443 (HTTPS), 53 (DNS) |
| Peak traffic hours | 10 AM–12 PM, 2 PM–4 PM |
| Average flow duration | 15–45 seconds |
| Ratio of outbound to inbound | ~0.2 (employees mostly download) |
Now suppose at 2 AM the outbound bytes spike to 15 GB in one hour, flows are short (1–2 seconds), and destinations are unusual IP ranges in Eastern Europe. This single 2 AM window deviates sharply from the established profile — a red flag for possible data exfiltration or a compromised machine communicating with a command-and-control server.
Sense-check: The anomaly is detected purely from metadata (sizes, timing, destinations) — no packet content inspection needed. This is exactly why profiling works even on encrypted traffic.
Assumptions & Scope: Traffic profiling works well when:
- Traffic patterns are relatively stable — the organization's work habits do not change drastically day to day.
- Sufficient historical data exists — you need at least a few days to weeks of baseline data to build a reliable profile.
- The network is not deliberately randomized — some advanced attackers use traffic shaping to mimic normal patterns, defeating simple profiling.
Profiling breaks down when:
- The organization has highly variable traffic — a media company uploading large video files may have wildly different daily patterns.
- New applications are deployed frequently — the "normal" baseline keeps shifting.
- Encrypted traffic uses padding or randomization — TLS 1.3 and protocols like QUIC can obscure packet sizes and timing.
Real-world: Traffic profiling connects to real-world scenarios like detecting unusual activity on organizational networks, identifying compromised devices, and distinguishing benign from malicious traffic patterns. Tools like Cisco Stealthwatch, Darktrace, and Vectra AI use traffic profiling as a core detection mechanism.
Recap + Bridge: Every network has a rhythm. Traffic profiling captures that rhythm as a statistical baseline. The next challenge: what happens when encryption hides the content of the traffic, and you can only observe the shape of the flow? That is the encryption challenge we turn to next.
12.2 The Encryption Challenge
12.2.1 The Shift from Plaintext to Encrypted Internet
Hook: What happens to security when you can no longer see what is inside the packets? For decades, security tools could read every byte on the wire. Then encryption became nearly universal — and those tools went blind overnight.
Historically, web traffic used HTTP (port 80), FTP, and Telnet — all insecure protocols where anyone on the wire could inspect every packet, email, or file transfer. Deep packet inspection (DPI) worked perfectly in this era because all content was visible in plaintext. A security tool could scan packet payloads for malware signatures, keywords, or suspicious patterns with complete visibility.
Intuition + Analogy: Think of the pre-encryption internet as a postal system where all letters are sent on postcards — anyone handling the postcard can read the message. Encryption turns every postcard into a sealed, opaque envelope. The postal service (network infrastructure) still sees the sender, receiver, size, and timing of each letter, but the contents are hidden.
Around 2013–2014, several forces converged to drive universal encryption:
- Privacy concerns: Snowden revelations and WikiLeaks raised public awareness about surveillance.
- Google's influence: Google began penalizing HTTP sites in search rankings, effectively making HTTPS a requirement for visibility. As one quip goes, "the best place to hide a dead body is Google's second page" — no one looks there, and HTTP sites increasingly ended up there.
- Let's Encrypt: This initiative provided free SSL/TLS certificates linked to domain registrations, removing the cost barrier. They have provided certificates for over 700 million websites.
- Regulatory pressure: GDPR in Europe, CCPA in California, and similar regulations mandated privacy protections.
- Browser security warnings: Modern browsers began flagging sites without valid certificates as insecure.
The result: by 2026, approximately 95% of internet traffic uses TLS 1.3, making the vast majority of traffic opaque to traditional inspection methods.
Real-world: The paper "The Cost of S in HTTPS" (2014, available on ACM) documents this transition and its implications for network security.
Pitfalls:
- Assuming encryption means security: Encryption hides content, but it does not hide metadata. Traffic patterns, timing, and flow characteristics remain visible.
- Thinking DPI can be "upgraded" to handle encryption: Deep packet inspection fundamentally cannot work on encrypted payloads without breaking the encryption (which defeats the purpose).
- Ignoring the encryption handshake metadata: The TLS handshake itself reveals valuable information — cipher suites, extensions, and client capabilities — which can be used for fingerprinting.
12.2.2 Impact on Security Operations
When traffic became encrypted, traditional security approaches broke down:
- Deep packet inspection became ineffective — you cannot inspect what you cannot read.
- Signature-based detection — regular expressions and pattern matching on packet payloads — stopped working for encrypted flows.
- Traditional IDS/IPS systems that relied on inspecting packet contents were rendered largely blind.
The fundamental shift: While encryption hides content, it cannot hide behavior. Machine learning analyzes how traffic moves — packet sizes, timing, flow patterns — not what it contains. This is the shift from content-based to metadata-based security analysis.
The metadata that remains visible includes:
- Packet sizes — how many bytes per packet
- Timing — inter-arrival times between packets, flow duration
- Direction — which side is sending, the ratio of inbound to outbound
- Flow patterns — connection frequency, destination diversity
- TLS handshake fields — cipher suites, extensions, SNI (Server Name Indication)
Worked Example: Consider two encrypted traffic flows to the same destination IP:
Flow A (legitimate web browsing):
- Packet sizes: varied (100–1500 bytes)
- Timing: bursty (request-response pattern)
- Duration: 30–120 seconds
- Direction: mostly inbound (downloading web pages)
Flow B (data exfiltration):
- Packet sizes: uniform (~1400 bytes, maxing out MTU)
- Timing: steady stream (continuous upload)
- Duration: hours
- Direction: mostly outbound (sending data out)
Both flows are encrypted with TLS 1.3. A DPI tool sees only ciphertext in both cases. But the metadata patterns are completely different — Flow B looks like a steady, long-duration upload, which is unusual for typical browsing. An ML model trained on these features can flag Flow B as anomalous without ever decrypting a single packet.
Sense-check: The anomaly is detected purely from flow characteristics, not content. This is the core insight that makes encrypted traffic analysis possible.
Analogy: Imagine airport security is not allowed to inspect bags. All you can observe is who is traveling from A to B, how often, what size bags they carry, and when. Only this metadata is available — but patterns in that metadata can still reveal anomalies. A passenger who flies to the same city every day carrying a large, heavy bag is behaving differently from a typical traveler — even though you never opened the bag.
12.2.3 TLS Protocol Evolution and the SSL Proxy Story
TLS Background: Transport Layer Security (TLS) is a cryptographic protocol that provides confidentiality and integrity for network communications. It operates above TCP (in the OSI model, at the session/presentation layer) and consists of two main components:
- The Handshake Protocol — negotiates cryptographic parameters, authenticates the server (and optionally the client), and establishes session keys.
- The Record Protocol — encrypts and authenticates application data using the negotiated keys.
TLS evolved from SSL (Secure Sockets Layer). The versions relevant to this lecture are TLS 1.1, TLS 1.2, and TLS 1.3.
The TLS protocol evolved through several versions, each with security implications:
TLS 1.1 and TLS 1.2 (one-way authentication): When a browser connected to a server (e.g., Gmail over HTTPS), the browser authenticated the server using pre-installed certificates. However, the server did not authenticate the browser. This asymmetry was exploited by SSL proxies.
How the SSL proxy worked (TLS ≤ 1.2):
- An organization would install a trusted certificate on each employee's laptop.
- All traffic from the laptop would route through the organization's SSL proxy.
- The proxy would terminate the HTTPS connection — the browser thought it was talking to Gmail, but it was talking to the proxy.
- The proxy would decrypt the traffic, inspect it, re-encrypt it, and forward it to the actual Gmail server.
- Gmail thought the proxy was the browser.
This man-in-the-middle approach worked because only the client authenticated the server, not vice versa. The proxy could present any certificate to the browser (including one signed by the organization's own CA), and the browser would accept it because the organization had installed that CA as trusted.
Key Intuition: TLS 1.3 (mutual authentication): TLS 1.3 closed this loophole by requiring the server to also authenticate the client. The proxy can no longer fool both sides — it cannot present a valid client certificate to Gmail because it does not have one. The SSL proxy approach collapsed.
In TLS 1.3, the handshake includes a CertificateVerify message that proves possession of the private key corresponding to the client certificate. A proxy that does not have the legitimate client's private key cannot forge this proof. This is why mutual authentication defeats man-in-the-middle proxies.
Student Q&A — Privacy and home networks:
Q: If the email was sent from a private home connection, then we don't have this intermediate SSL proxy layer?
A: Correct. When working from home without a VPN, you connect directly to the cloud or Gmail server. However, if your organization uses an endpoint agent like Zscaler or a cloud access security broker, all traffic may still route through the company's security infrastructure even from home. The agent connects to your office account in the cloud, where company policies are applied before traffic reaches its destination.
Student Q&A — Office machine without VPN:
Q: If we work from home with the office machine and are not connected to the VPN, is the SSL proxy still in place?
A: No, it cannot be — you are directly talking to the cloud. But if there is an endpoint agent (Zscaler or similar) running on the laptop, the traffic may still go through the company's security server first, where policies are applied, before reaching the destination.
Student Q&A — Browser certificates vs. SSL proxy:
Q: How do we decide whether the browser certificate is valid or whether we go with the SSL proxy?
A: Browsers come pre-installed with certificates from root certificate authorities (Verizon, Comodo, etc.). You can view these in Chrome under Settings → Privacy and Security → certificates. The browser uses these to verify server identity. The SSL proxy must mimic thousands of endpoint certificates, which is not scalable. With TLS 1.3's mutual authentication, the proxy approach fundamentally fails.
Student Q&A — C2C server and SSL proxy compromise:
Q: Could a C2C server set up a valid certificate inside the SSL proxy, allowing the client to communicate without being blocked?
A: Yes, these attacks happen. The SSL proxy itself can be compromised with a fraudulent certificate — this is essentially a man-in-the-middle attack. Machine learning helps detect such anomalies regardless of whether they involve MITM proxies or not.
Pitfalls:
- Confusing TLS termination with TLS interception: TLS termination (by a legitimate proxy) decrypts traffic for inspection. TLS interception (by an attacker) is a man-in-the-middle attack. The technical mechanism is similar; the authorization is different.
- Assuming TLS 1.3 makes all inspection impossible: TLS 1.3 defeats SSL proxies that rely on one-way authentication, but endpoint agents can still inspect traffic before encryption (on the host) or after decryption (at the application layer).
- Overlooking certificate transparency logs: Even with encryption, certificate issuance is public. Tools like crt.sh allow monitoring for suspicious certificate registrations, which can reveal phishing or command-and-control infrastructure.
Recap + Bridge: Encryption made deep packet inspection obsolete for most traffic. But the metadata — packet sizes, timing, flow patterns, TLS handshake fields — remains visible. The next question: how do we use that metadata to identify specific applications, malware, or users? That leads us to traffic fingerprinting.
12.3 Traffic Fingerprinting — JA3 and JA4+
12.3.1 What Is Fingerprinting?
Hook: If you cannot read the letter inside the envelope, can you still tell who sent it? Network fingerprinting says yes — by studying the envelope itself (the TLS handshake), you can identify the sender with surprising accuracy.
Fingerprinting means uniquely identifying an artifact by its characteristics. In forensics, fingerprints identify suspects because they are unique. Applied to network traffic, fingerprinting asks: is there a way to uniquely characterize traffic so you can distinguish good from bad, allowed applications from denied ones, or one type of traffic from another?
Intuition + Analogy: Think of TLS fingerprinting like recognizing a person by their handshake style. You cannot see their face (encrypted content), but you can observe how they grip, how firmly they squeeze, whether they use one hand or two, and how long they hold. Different applications and malware have different "handshake styles" — the specific cipher suites they support, the extensions they include, and the order they present them. These details form a fingerprint.
Since you cannot inspect encrypted packet contents, you must work with what is visible: the TLS handshake metadata, packet sizes, timing, and flow characteristics. The goal is to encode these raw parameters into a structured, reproducible string — a fingerprint — that serves as a signature.
Formalize: A TLS fingerprint is a hash or structured string derived from the fields in the TLS ClientHello message. The ClientHello is the first message in a TLS handshake, sent by the client to the server. It contains:
- SSL/TLS version — the highest version the client supports
- Cipher suites — the list of encryption algorithms the client can use, in preference order
- Extensions — additional features like SNI (Server Name Indication), supported groups, signature algorithms
- Elliptic curves — the curves the client supports for key exchange
- Elliptic curve point formats — how curve points are encoded
These fields are concatenated in a defined order and hashed to produce a compact fingerprint string.
12.3.2 JA3 Fingerprinting
JA3 was the first widely adopted standard for TLS fingerprinting, developed by Salesforce in 2017. The method is straightforward:
- Collect specific fields from the TLS handshake: SSL version, cipher suites, extensions, elliptic curves, and elliptic curve point formats.
- Concatenate them in a defined order (comma-separated within each field, comma-delimited between fields).
- Compute an MD5 hash of the concatenation.
The result is a JA3 fingerprint string. For example, the TrickBot malware has a specific JA3 fingerprint that can be used to identify its traffic.
Worked Example: Suppose a TLS ClientHello contains:
- SSL Version: TLS 1.2 (0x0303)
- Cipher Suites: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
- Extensions: server_name, status_request, supported_groups, signature_algorithms
- Elliptic Curves: x25519, secp256r1, secp384r1
- Point Formats: uncompressed
The JA3 string would be: 771,49199-49200,0-5-13-11-10,29-23-24,0
Then the MD5 hash of this string produces the JA3 fingerprint (e.g., e7d705a3286e19ea42f587b344ee6865).
Sense-check: Different applications (Chrome, Firefox, curl, malware) will have different ClientHello fields, producing different JA3 fingerprints. This allows identification even when the content is encrypted.
Limitations of JA3:
- Brittleness: Even a subtle change in traffic parameters completely changes the hash. A single different cipher suite or extension produces an entirely different fingerprint. This makes JA3 fragile — software updates can change the fingerprint.
- MD5 is broken: MD5 is cryptographically weak and should not be used for security-sensitive hashing. While JA3 uses MD5 for identification (not security), using a broken hash function is still poor practice.
- Ordering ambiguity: Different implementations might order fields differently, producing different hashes for the same traffic. There is no strict standard for field ordering.
- Library: There is a Python library for JA3 extraction, though the standard itself is now considered obsolete.
Exam note: You should be able to distinguish JA3 from JA4+ and understand how each is computed and what the fingerprint looks like. You will not be asked to compute a SHA hash by hand, but conceptual understanding is required.
12.3.3 JA4+ Fingerprinting
JA4+ is a significantly improved, modular fingerprinting standard designed to address JA3's limitations. It was developed by John Althouse and released in 2023 as a successor to JA3.
Formalize — JA4+ Structure: A JA4+ fingerprint is composed of multiple underscore-separated segments, each encoding a specific aspect of the traffic. The format is:
[Protocol][Version][SNI][CipherCount][CipherHash]_[Extensions][SignatureAlgorithms]_[MiscFields]
For example: T13_SNI_9e7d705a3286e19ea...
Each component is interpretable:
T— Protocol: TLS13— Version: TLS 1.3S— SNI present: yesN— Next Protocol Negotiation: noI— (reserved)9— Cipher suite count: 9e7d705a3286e19ea...— Hash of cipher suites (first 12 characters of SHA-256)
Worked Example — JA4+ vs JA3 Comparison:
Consider the same TLS ClientHello:
| Field | Value |
|---|---|
| Protocol | TLS |
| Version | 1.3 |
| SNI | example.com |
| Cipher Suites | 9 suites |
| Extensions | 12 extensions |
JA3: 771,49199-49200-...,0-5-13-11-10,29-23-24,0 → MD5 → e7d705a3286e19ea42f587b344ee6865
JA4+: T13_SNI_9e7d705a3286e19ea_0-5-13-11-10_29-23-24
The key difference: JA3 produces a single opaque hash. JA4+ produces a structured, human-readable string where each segment is interpretable. You can filter by protocol version, count cipher suites, or check for SNI — all without decoding the hash.
Sense-check: JA4+ is modular — you can compare fingerprints by individual components, not just the whole hash. This makes it far more useful for security analysis and threat hunting.
Applications of JA4+:
- Browser identification — determine which browser is being used (Chrome, Firefox, Safari have distinct fingerprints)
- Malware pattern detection — identify known malicious traffic signatures (e.g., TrickBot, Emotet have characteristic JA4+ fingerprints)
- User tracking and cross-session identity linking — researchers demonstrated that over 150,000 users across two years could be uniquely fingerprinted using JA4+, eliminating 84–95% of anonymity
- Traffic classification — feed fingerprints into ML models for binary or multi-class classification (malware vs. benign, application type, etc.)
- Network policy enforcement — block traffic from unauthorized applications based on their fingerprint
Pitfalls:
- Confusing JA3 and JA4+: JA3 uses MD5 and produces an opaque hash. JA4+ uses SHA-256 and produces a structured, interpretable string. They are not interchangeable.
- Assuming fingerprints are immutable: Software updates, TLS library changes, or configuration changes can alter a client's fingerprint. Fingerprint databases need regular updating.
- Over-relying on fingerprints alone: Fingerprints identify the client software, not the user or intent. A Chrome fingerprint could be legitimate browsing or a malware-infected Chrome process. Fingerprints are best used as one feature among many in ML models.
Recap + Bridge: Traffic fingerprinting encodes TLS handshake characteristics into structured signatures. JA3 was the first standard but had limitations; JA4+ improved with modularity and readability. But fingerprints are just one type of feature. The next step is extracting richer flow-level features from raw packets — that is the domain of tools like CIC Flow Meter.
12.4 Flow Feature Extraction — CIC Flow Meter
12.4.1 From Raw Packets to Structured Features
Hook: Raw packet captures (pcap files) contain thousands of individual packets — analyzing them one by one is like reading every individual transaction in a bank statement to understand spending habits. You need to aggregate first. Flow feature extraction transforms raw packets into structured summaries that ML models can reason about.
Rather than working with raw packet captures, security analysts typically extract structured flow-level features first. A flow is a sequence of packets between the same source and destination (identified by the 5-tuple: source IP, destination IP, source port, destination port, protocol). The CIC Flow Meter (from the Canadian Institute for Cybersecurity, approximately 5–8 years old) is an open-source tool that processes pcap files and extracts standard bi-directional flow features.
Formalize — What CIC Flow Meter extracts: For each bidirectional flow, CIC Flow Meter computes:
- Flow duration — time between first and last packet
- Bytes per second — throughput (forward and backward)
- Packet counts — number of packets in each direction
- Inter-arrival times — mean, standard deviation, max, min of time gaps between packets
- Packet length statistics — mean, std, max, min of packet sizes in each direction
- Flag counts — number of SYN, ACK, PSH, URG, FIN, RST flags
- Header lengths — total bytes in packet headers
- Bulk features — bytes/packets per second in bulk transfers
- Subflow features — statistics for sub-flows (time-windowed segments)
- Active/Idle times — periods of activity and silence
These features are computed separately for forward (source→destination) and backward (destination→source) directions, producing a rich feature vector for each flow.
Worked Example: Consider a pcap file containing a TCP session to a web server. After processing with CIC Flow Meter, one flow might produce:
| Feature | Value |
|---|---|
| Flow Duration | 12.3 seconds |
| Total Fwd Packets | 15 |
| Total Bwd Packets | 12 |
| Fwd Bytes/s | 45,200 |
| Bwd Bytes/s | 128,500 |
| Fwd Packet Length Mean | 320 bytes |
| Bwd Packet Length Mean | 1,420 bytes |
| Fwd IAT Mean | 0.82 seconds |
| Bwd IAT Mean | 0.15 seconds |
| SYN Flag Count | 1 |
| ACK Flag Count | 26 |
This 40+ dimensional feature vector is what gets fed into ML models. The model does not see raw packets — it sees this structured summary.
Sense-check: The features capture behavior (how much data, how fast, in what pattern) without needing to decrypt the content. This is why CIC Flow Meter works on encrypted traffic.
Assumptions & Scope: CIC Flow Meter works well when:
- Traffic is TCP or UDP — the tool is designed for these protocols. Other protocols (ICMP, GRE) may not produce meaningful flows.
- Flows are well-defined — the 5-tuple must uniquely identify a conversation. NAT, load balancers, or VPNs can merge or split flows.
- Pcap files are complete — missing packets (due to capture drops) can skew statistics.
The tool has limitations:
- No content analysis — it cannot detect application-layer attacks (SQL injection, XSS) because it does not inspect payloads.
- Feature engineering is fixed — you cannot easily add custom features without modifying the source code.
- High-dimensional output — 40+ features per flow requires dimensionality reduction or feature selection for effective ML.
Note: Packet capture formats and TCP/IP protocols change very slowly, so tools like CIC Flow Meter remain relevant even after several years. You can add features, but the underlying capture structure and protocols rarely change. The pcap format (libpcap) has been stable since the 1990s.
Intuition + Analogy: Think of CIC Flow Meter as a traffic census taker. It stands at an intersection and records: how many cars passed, how fast they were going, what direction, how long between cars, and whether they stopped or turned. It does not look inside the cars. But from this census data, you can tell if it is a normal rush hour or a high-speed chase.
Real-world: CIC Flow Meter is available as an open-source tool. You can capture traffic with Wireshark, export to pcap format, and feed it to CIC Flow Meter for feature extraction. The Canadian Institute for Cybersecurity also provides labeled datasets (CICIDS2017, CICIDS2018) built using this tool for benchmarking ML models.
Recap + Bridge: CIC Flow Meter transforms raw packets into structured feature vectors that capture flow behavior without inspecting content. But traffic analysis is not just about network flows — there is a human dimension. Users have patterns too, and when those patterns deviate, it may signal compromise. That brings us to User Entity Behavior Analytics.
12.5 User Entity Behavior Analytics (UEBA)
12.5.1 The Missing User Dimension
Hook: For decades, security teams watched the network, the servers, and the firewalls — but they ignored the users. Then they realized: the user is the weakest link. UEBA puts the user back in the center of security monitoring.
Traditional security operations — SIEM tools, firewall logs, packet captures — focused heavily on network and system artifacts while largely ignoring the user dimension. Only in the last five or six years has the industry recognized that the user is a critical missing piece. UEBA (User Entity Behavior Analytics) addresses this gap by analyzing user activity patterns alongside network and entity behavior.
Formalize — What UEBA is: UEBA is a security analytics approach that builds a behavioral baseline for each user (or entity — a device, application, or service) and detects deviations from that baseline. The core idea: instead of defining rules for what is "bad," learn what is "normal" for each user and flag when behavior changes.
UEBA draws on data sources including:
- Active Directory logs — login events, account lockouts, group membership changes
- Authentication logs — login success/failure, source IP, geolocation, device type
- Application access logs — which applications a user accesses, when, and how
- User activity sequences — the order and timing of actions (e.g., login → email → file download → logout)
- Network traffic metadata — connections made, destinations, data volumes
Intuition + Analogy: Think of UEBA like a credit card fraud detection system. Your bank knows your typical spending patterns — where you shop, how much you spend, what time of day. When a charge appears that does not match your pattern (a 5,000 dollar purchase in a country you have never visited), the system flags it. UEBA does the same for user behavior on a network: it learns your typical login times, applications, data access patterns, and alerts when something is off.
The goal is to pinpointedly identify when a user's machine or account is compromised so the security team can take targeted action. UEBA does not replace network-level detection — it complements it by adding the user context that network tools miss.
12.5.2 Key UEBA Use Cases
Worked Example — Impossible Travel: A user logs in from New York at 9:00 AM EST. At 9:15 AM EST, the same account logs in from Beijing. The physical distance is approximately 11,000 km. Even by the fastest commercial flight (12+ hours), this is impossible.
This was illustrated by a real case: an employee in New York appeared to work diligently at his desk every day from 9 to 6, but all his actual work — emails, code, everything — was being done by someone in China who had his RSA token. The employee had shipped his hardware to China and was paying someone a fraction of his salary to do his job while he watched videos all day.
Detection logic:
- Record the geolocation of every login event (from IP geolocation).
- Calculate the physical distance between consecutive logins.
- Calculate the time difference between consecutive logins.
- If
distance / time > maximum_possible_travel_speed, flag as impossible travel.
Sense-check: This works because physics constrains how fast a human can travel. Even with VPNs (which can make the IP appear in a different location), the pattern of rapid, geographically distant logins is suspicious.
Resource access anomalies: An engineer suddenly using an HR application with administrative access, or a user generating thousands of SSH connections when their normal pattern shows very few. These anomalies are detected by comparing current behavior to the user's historical baseline.
Worked Example — Peer Group Analysis: Consider a company with three departments: Engineering (10 people), Sales (8 people), and Finance (6 people). Normally:
- Engineers communicate mostly with other engineers (internal project collaboration)
- Salespeople communicate with clients and each other
- Finance communicates with everyone but in low volume
If a Finance user suddenly starts communicating extensively with an Engineer on a completely different project — especially if they have never interacted before — this deviation signals potential compromise. The peer group concept defines expected interaction clusters based on organizational structure, and deviations from these clusters are flagged.
Detection logic:
- Build a communication graph: nodes are users, edges are communication frequency.
- Cluster users into peer groups using graph clustering (e.g., community detection).
- For each user, track the ratio of intra-group to inter-group communication.
- If the ratio changes significantly (e.g., a Finance user starts talking 10× more to Engineering), flag the anomaly.
Sense-check: This catches insider threats and compromised accounts that behave normally within their own group but abnormally across groups.
Brute force detection: Discussed in detail in the class exercise below.
Pitfalls:
- Baseline drift: User behavior changes over time (new projects, role changes, seasonal patterns). The UEBA system must continuously update baselines, not use a static model.
- False positives from legitimate changes: An employee transferring departments will trigger peer group anomalies. The system needs mechanisms to handle legitimate role changes.
- Privacy concerns: UEBA monitors individual user behavior, which raises privacy issues. Organizations must balance security monitoring with employee privacy rights and comply with regulations like GDPR.
12.5.3 Interactive Exercise — Brute Force Attack and Defense
The class conducted a live attacker-defender exercise to explore brute force attack detection:
Setup: The application logs record login attempts (success/failure), source IP, and timestamps. The attacker (using Daniel Miessler's publicly available password list from GitHub, or a script generating alphanumeric combinations) tries passwords against a target application.
Defender strategies discussed:
- Count failed login attempts: Track consecutive failures within a time window. If failures exceed a threshold (e.g., 3 or 5), lock the account.
- Geolocation checking: If login attempts originate from geographically distant locations within a short time, flag as suspicious (impossible travel).
- Source device identification: Not just the application logs, but identify which device is attempting the login — is it a known corporate device or an unknown source?
- Account lockout with manager notification: After threshold failures, lock the account and trigger an email to the manager for re-enablement.
- CAPTCHA introduction: Add CAPTCHA challenges to distinguish human from automated attempts.
- Device quarantine: Cut off the attacking device from network access entirely.
- Time-based lockouts: Lock the account for 24–48 hours after repeated failures.
Attacker counter-strategies discussed:
- Distribute across multiple user IDs: Instead of hammering one account, spread attempts across many accounts to stay below per-account thresholds.
- Change source network: If the source IP is identified and blocked, switch to a different network.
- Wait out the lockout period: After a 24-hour freeze, resume attempts.
- Lateral movement: If one machine is quarantined, move to another machine on the network.
- Motivation shift: The attacker's goal may not be to gain access — simply locking out legitimate users disrupts productivity (denial of service through brute force).
Key insight from the exercise: Brute force detection is not a solved problem. It involves ongoing trade-offs between security and usability. Locking accounts after failures protects against attackers but also enables denial-of-service — an attacker can lock out every user simply by triggering failures. The defender must balance thresholds, monitoring, and response mechanisms.
This is a classic security-usability trade-off: more aggressive locking = better security against brute force, but also more risk of self-inflicted denial of service.
Student Q&A — Identity provider solutions:
Q: What if we use an identity provider like Azure AD or Okta? They have built-in identity protection using AI to detect and block threats automatically.
A: You have shifted the problem from your application to the identity provider. The identity provider still needs to solve the same problem — detecting brute force, impossible travel, and anomalous access. Tools like Microsoft Defender for Identity and conditional access policies are the mechanisms, but the underlying detection challenges remain. The identity provider has a broader view (seeing authentication across many applications), which can improve detection, but the fundamental trade-offs between security and usability persist.
Exam note: Expect questions on UEBA in the final exam. Understand impossible travel, peer group analysis, and the trade-offs in brute force defense. Know the data sources UEBA uses and how behavioral baselines are built.
Recap + Bridge: UEBA adds the missing user dimension to security monitoring by building behavioral baselines and detecting deviations. Combined with traffic profiling, fingerprinting, and flow feature extraction, it forms a comprehensive toolkit for detecting threats in encrypted networks. The lecture also touched on several additional topics that round out the picture — malware variants, IoT challenges, and certificate transparency.
12.6 Additional Topics Mentioned
12.6.1 Malware Variants and Other Challenges
Hook: The security landscape is not static — attackers evolve their tools just as defenders evolve their defenses. Understanding the categories of threats helps you choose the right detection approach.
Polymorphic vs. metamorphic malware: Polymorphic malware changes its code while maintaining similar behavior. Metamorphic malware changes both its code and behavior. This distinction matters for detection — signature-based methods struggle with both, but behavioral analysis can sometimes catch polymorphic variants.
Formalize — Malware Evolution:
| Type | Code Changes? | Behavior Changes? | Detection Challenge |
|---|---|---|---|
| Static Malware | No | No | Easy — signature matching works |
| Polymorphic | Yes (encryption, packing) | No | Harder — but behavioral signatures still work |
| Metamorphic | Yes (rewrites itself) | Yes | Hardest — must use advanced behavioral analysis or ML |
Polymorphic malware uses encryption or packing to change its binary representation each time it executes, but the underlying behavior (what it does) remains the same. Metamorphic malware goes further — it rewrites its own code, potentially changing its behavior as well. This makes signature-based detection essentially useless against metamorphic variants.
Zero-day exploits: Attacks exploiting unknown vulnerabilities where no signature exists. Zero-days are particularly dangerous because there is no patch and no signature — the attack exploits a vulnerability that the software vendor does not yet know about. Detection relies entirely on behavioral analysis and anomaly detection.
IoT devices: Use unknown or proprietary protocols, adding complexity to traffic analysis. Many IoT devices communicate over custom protocols that standard network analysis tools do not understand. Additionally, IoT devices often have weak security (default passwords, unpatched firmware), making them attractive targets for botnets.
High-speed networks: Deep packet inspection is too slow for 100 Gbps+ lines, further motivating metadata-based approaches. At 100 Gbps, a single link carries approximately 148 million packets per second — inspecting each packet's payload in real time is computationally infeasible. This is another reason why metadata-based approaches (profiling, fingerprinting, flow features) are essential for modern network security.
Export controls on encryption: Historically, encryption algorithms required export licenses. Some governments debated banning encryption because criminals were using it — analogous to banning physics research because it could lead to nuclear weapons. Encryption is a tool that can be used for good (privacy, security) or bad (hiding criminal activity). Banning it would harm legitimate users while criminals would simply use illegal implementations.
Certificate transparency logs: Public logs of all issued SSL certificates. You can subscribe to these logs and perform analysis (e.g., n-gram analysis on domain names) to detect phishing sites or suspicious certificate registrations. The tool crt.sh was mentioned for searching these logs, though it was down at the time of the lecture. Certificate transparency is a critical defense against certificate misissuance and man-in-the-middle attacks using fraudulent certificates.
Squid proxy: An open-source proxy server that can be used for traffic inspection and access control. The lecturer shared a personal anecdote about using it 20 years ago — their boss would block internet access via Squid if work was not completed by the previous day. Squid is still used today in enterprise environments for web filtering, caching, and access control.
Zscaler and cloud access security brokers: Commercial solutions that act as web application proxies, applying company policies to all traffic even when employees work from home. These solutions route all traffic through the provider's cloud, where security policies are applied before traffic reaches its destination. This is the modern evolution of the enterprise proxy — moving from on-premises (Squid) to cloud-based (Zscaler).
Pitfalls:
- Confusing polymorphic and metamorphic malware: Polymorphic changes code but not behavior; metamorphic changes both. The detection strategy differs for each.
- Assuming zero-days are undetectable: Zero-days exploit unknown vulnerabilities, but the behavior of the exploit (unusual network traffic, abnormal process activity) can still be detected by behavioral analysis.
- Overlooking IoT as a security risk: IoT devices are often treated as "set and forget" — but they are network-connected computers that need security monitoring like any other endpoint.
Recap: The additional topics covered in this lecture round out the picture of modern network security challenges. Malware is evolving (polymorphic, metamorphic), IoT adds new attack surfaces, high-speed networks make DPI impractical, and certificate transparency provides a public accountability mechanism for SSL certificates. Together with the core topics of traffic profiling, fingerprinting, flow features, and UEBA, these concepts form the foundation of ML-based network security in the encrypted era.
Exam Guidance Summary
Exam note: The following topics are examinable for Lecture 12:
- UEBA questions: Expect a question on User Entity Behavior Analytics in the final exam. Understand the concept, use cases (impossible travel, peer group analysis, resource access anomalies), and the data sources involved.
- JA3 vs. JA4+: You should be able to distinguish the two fingerprinting methods, understand how each is computed, and recognize what the fingerprints look like. No hand-computation required.
- Brute force detection trade-offs: Understand the attacker-defender dynamics, the risk of account lockout enabling denial of service, and the role of identity providers.
- Encryption impact on security: Understand why traditional DPI fails with encryption and how ML-based approaches work on metadata instead of content.
- TLS 1.3: Understand why mutual authentication defeats SSL proxies.
- Cyber Kill Chain framework: Expect applied thinking questions — not "what is the algorithm" but "how would you improve it" or "how would you apply it." High-level understanding of its design is required.
- Slides: The intentionally sparse slides are by design. Focus on understanding concepts from lecture discussion rather than memorizing slide content.
- Watermarked PPTs: Will be shared closer to the exam. Do not print early in case the watermark symbol changes.
Key Industry Applications
Real-world tools and services mentioned in this lecture:
- Let's Encrypt: Free SSL certificates that democratized HTTPS adoption for 700 million+ websites. Issued by the Internet Security Research Group (ISRG), it removed the cost barrier to HTTPS, contributing to the encryption shift discussed in Section 12.2.
- Zscaler: Commercial cloud security platform acting as a web application proxy with policy enforcement. Routes all traffic through Zscaler's cloud, where security policies are applied — the modern evolution of the enterprise proxy.
- Squid proxy: Open-source proxy server for traffic inspection and access control. Still used in enterprise environments for web filtering, caching, and access control.
- CIC Flow Meter: Open-source tool from the Canadian Institute for Cybersecurity for extracting flow-level features from packet captures. Used to build labeled datasets (CICIDS2017, CICIDS2018) for benchmarking ML models.
- Wireshark: Free network packet analyzer for capturing and inspecting traffic. The industry standard for packet capture and analysis.
- Microsoft Defender for Identity / Azure AD Identity Protection: AI-powered identity threat detection using sign-in logs and conditional access policies. Implements UEBA concepts (impossible travel, anomalous access) at enterprise scale.
- Okta / Azure AD: Identity providers with built-in brute force and anomaly detection. Shift the detection challenge from individual applications to the identity provider layer.
- Certificate Transparency Logs (crt.sh): Public infrastructure for monitoring SSL certificate issuance. Enables detection of fraudulent certificates and phishing domains through public accountability.
AMTCS Lecture 12 notes · Network Traffic Profiling and Encrypted Traffic Analysis
Sections Breakdown
Motivation, concept, and observable features of traffic profiling for anomaly detection
How universal encryption rendered DPI ineffective and the shift to metadata-based analysis
TLS handshake fingerprinting standards for identifying encrypted traffic
Extracting structured flow-level features from pcap files for ML analysis
Behavioral baselines, impossible travel, peer group analysis, and brute force defense
Polymorphic/metamorphic malware, IoT challenges, certificate transparency, and cloud proxies
Exam Revision Notes
Below is the distilled, exam-ready core. Every entry comes from the full explanation above. Use this section for rapid review; return to the main notes when a point needs more context.
Network Traffic Profiling
Must-know: Traffic profiling encodes observable characteristics (size, timing, direction, protocol metadata) into structured representations for anomaly detection, working even on encrypted traffic.
⚠️ Top pitfall: Assuming traffic profiling requires inspecting packet contents — it works purely on metadata and flow statistics.
Self-check: What observable features are used in traffic profiling when packet contents are encrypted?
Connects to: The Encryption Challenge, CIC Flow Meter
The Encryption Challenge
Must-know: TLS 1.3 mutual authentication defeats SSL proxies because the proxy cannot present a valid client certificate. Encryption hides content but not metadata — ML analyzes packet sizes, timing, and flow patterns instead.
⚠️ Top pitfall: Assuming DPI can be upgraded to handle encryption — it fundamentally cannot work on encrypted payloads without breaking the encryption.
Self-check: Why does TLS 1.3 defeat the SSL proxy approach that worked with TLS 1.2?
Connects to: Network Traffic Profiling, Traffic Fingerprinting
Traffic Fingerprinting — JA3 and JA4+
Must-know: JA3 concatenates TLS ClientHello fields and hashes with MD5. JA4+ produces modular, interpretable segments with SHA-256. JA4+ is the current industry standard for encrypted traffic fingerprinting.
⚠️ Top pitfall: Confusing JA3 and JA4+ — JA3 uses MD5 and produces opaque hashes; JA4+ uses SHA-256 and produces structured strings with interpretable segments.
Self-check: What are two key differences between JA3 and JA4+ fingerprinting?
Connects to: The Encryption Challenge, CIC Flow Meter
Flow Feature Extraction — CIC Flow Meter
Must-know: CIC Flow Meter extracts 40+ features per bidirectional flow from pcap files, capturing behavior (duration, throughput, timing, flags) without decrypting content. Works on encrypted traffic.
⚠️ Top pitfall: Assuming CIC Flow Meter can detect application-layer attacks — it only extracts flow-level metadata, not packet payloads.
Self-check: Name three categories of features extracted by CIC Flow Meter.
Connects to: Network Traffic Profiling, Traffic Fingerprinting, UEBA
User Entity Behavior Analytics (UEBA)
Must-know: UEBA detects compromised accounts by building behavioral baselines and flagging deviations: impossible travel (geographically distant logins in impossible time), peer group analysis (unusual cross-group communication), and resource access anomalies. Brute force defense involves security-usability trade-offs — account lockout can itself become a DoS vector.
⚠️ Top pitfall: Ignoring that brute force account lockout can itself be weaponized as a denial-of-service attack.
Self-check: What is impossible travel in UEBA, and how is it detected?
Connects to: Network Traffic Profiling, The Encryption Challenge, CIC Flow Meter
Additional Topics
Must-know: Polymorphic malware changes code but not behavior; metamorphic changes both. Zero-days exploit unknown vulnerabilities but behavioral analysis can still detect the exploit. High-speed networks (100 Gbps+) make DPI infeasible, motivating metadata-based approaches.
⚠️ Top pitfall: Confusing polymorphic and metamorphic malware — polymorphic changes code but keeps behavior; metamorphic changes both.
Self-check: What is the difference between polymorphic and metamorphic malware?
Connects to: The Encryption Challenge, Traffic Fingerprinting
Was this lecture useful?
BitsNotes AI Assistant
Subject Notes AssistantConfigure AI Chat
Choose how to access the chatbotSigned in as
Powered by BitsNotes — 20 messages per day. No API key needed. Want unlimited access? Use "Bring Your Own Key" mode.
Sign in to use AI Chat
Get 20 free AI messages per day to ask questions about your lecture notes. Sign in with Google or GitHub — it takes 5 seconds.
Sign In to BitsNotesSwitch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.