Overlapping SGD, Quantized SGD, and Federated Learning

# Overlapping SGD, Quantized SGD, and Federated Learning

Two SGD communication tricks still sit on the table after the earlier distributed training variants: overlapping SGD and quantized SGD. Overlapping mixes compute with network traffic so a GPU does not sit idle. Quantization shrinks the bit-width of numbers so those messages (and often the math itself) get cheaper. Gradient buckets sit between those two ideas: they group many gradients into one packet so overlap can start as soon as a chunk is ready.

After these SGD variants, the story shifts to federated learning (FL): train where the data already lives, share updates instead of raw records, and then face the new attack that reconstructs training-like samples from those shared weights. The second half of the session is the start of the federated half of the course: definition, a voice-assistant loop, FedAvg arithmetic, hospital silos, and why “I only sent numbers” is not a privacy proof.

Keep the two halves separate in your notes. Overlap, quantization, and buckets are still lab-style distributed training: data can be sharded by a trainer who already owns it. Federated learning is the different setting where the trainer is not allowed to own the raw records. The same averaging algebra will return later as FedAvg, but the threat model changes.

9.1 Overlapping SGD: Mixing Compute with Communication

Each GPU in a pipelined or partitioned model has two jobs, not one. The first job is computation: in the forward pass, input activations enter the local layer (or the local slice of the model), the GPU runs the layer math, and activations leave toward the next stage. The second job is communication: in the backward pass, gradients must travel back toward earlier layers, and in the forward pass the activations themselves must travel toward later layers.

The design goal is to intermix computation and communication on the same machine so neither side idles while the other runs. Think of a chain of GPUs that each own one layer or a small block of layers. Data walks forward through GPU 1, GPU 2, GPU 3, and so on. Gradients walk backward along the same chain. The naive habit is to finish one of those jobs completely before starting the other. Overlapping SGD refuses that wait.

A kitchen picture helps. The cook (compute) chops the next plate while the waiter (the interconnect) already carries the previous plate. The kitchen does not wait for the waiter to return empty-handed before the knife moves. The analogy breaks when the waiter's trip is longer than the chopping: then the cook still finishes first and the wait is the network, not the math.

Training uses many batches. Each batch must be handed to the right GPUs, turned into gradients, and used to update layer weights. Layer 1 should update before (or at least independently of) later layers in the backward order, because the backward wave starts at the loss and moves toward the input. At the same time, the next forward wave already wants those GPUs. If a device waits until it has both finished sending every gradient and finished applying every update, the pipeline stalls.

Standard treatments of data-parallel training make the same point in a different costume. After each iteration you still have to aggregate gradients and then push updated weights. Those two communication steps cannot collapse into one instant: GPUs wait for the sum of both transfer times unless later work is allowed to start early. Overlap is the pipeline version of “do not add those waits if the hardware can hide one of them.”

9.1.1 Forward Compute and Backward Send on the Same GPU

In the forward process, the GPU performs operations on the data it currently holds. That data may be a full model or a subset, such as a single layer. The local layer produces an output tensor. That tensor is the input of the next stage. After the last stage produces a prediction, backward computation starts. Gradients with respect to activations and weights are formed and then sent back along the pipeline: back to the previous GPU, then the one before that, and so on.

Each GPU then faces traffic in two directions:

• Activations toward the “upper” (later) layers in the forward pass.
• Gradients toward the “down” (earlier) layers in the backward pass.

The verbal explanation of the two tasks is: the input goes through, the GPU performs the computation and gives the output in the forward pass, and in the backward pass it has to communicate, passing through the gradients. Those two tasks do not have to be a strict sequence. Once a layer has produced its forward output, it can start sending that output while it also prepares or continues other local work. Once a later layer has gradients ready for an earlier layer, it can start sending those gradients while it still computes remaining local backward terms.

Name the tensors so the two jobs stay distinct. Let $x^{(\ell )}$ be the activation arriving at stage $\ell$, and let $h^{(\ell )}=f_{\ell }(x^{(\ell )};{\theta }^{(\ell )})$ be the activation that stage $\ell$ produces with local weights ${\theta }^{(\ell )}$. The send of $h^{(\ell )}$ toward stage $\ell +1$ can start as soon as $h^{(\ell )}$ exists. In the backward pass, let $g^{(\ell )}=\partial L/\partial h^{(\ell )}$ be the incoming activation gradient. Local weight gradients $\partial L/\partial {\theta }^{(\ell )}$ can be formed while $g^{(\ell -1)}$ is already on the wire toward stage $\ell -1$.

9.1.2 Staggered Compute-Then-Send Timeline

A classroom demo walks layer by layer. Layer 5 computes. Then it sends. The next layer computes while the previous layer’s send may still be in flight. Then that next layer sends. The pattern is compute, send, compute, send, compute, send. When one layer has finished computing and has already handed its tensor to the next layer, it can be in a sending state while the neighbor is in a computing state. That staggered overlap is the whole trick.

Picture a Gantt chart. The horizontal axis is time in milliseconds. The vertical axis is stage index (layer 1 at the bottom, later layers above). Each stage has two bars: a compute bar and a communicate bar. In the ordered picture the communicate bar starts only after the compute bar ends, so the bars sit end to end. In the overlap picture the communicate bar of stage $\ell$ slides under the compute bar of stage $\ell +1$. The landmark to watch is the first moment a communicate bar sits under a neighbor's compute bar: that is where idle time died.

The saving is the hidden wait. If send is fully nested under compute (or the reverse), you drop the smaller term from the critical path. If they only partially overlap, you still beat the strict sum. Let $T_{\text{exposed}}$ be the leftover that did not hide. Then

$$T_{\text{partial}}=\max (T_{\text{compute}},T_{\text{communicate}})+T_{\text{exposed}}$$

and $T_{\text{ordered}}\ge T_{\text{partial}}\ge T_{\text{overlap}}$. When you overlap the computation and the communication steps, you cut the idle times, which you cannot cut in the non-overlap case.

9.1.3 Worked Timeline: Several Batches on One Machine

Take a short pipeline of layers and several mini-batches already waiting in memory. Batch 1 enters layer 1. Layer 1 computes activations and starts sending them to layer 2. Layer 2 computes while layer 1 may already be receiving or preparing the next slice. After the forward wave reaches the last layer, backward gradients form and travel back: sending back, sending back, sending back. While those gradients for batch 1 are still on the wire toward earlier layers, batch 2 can already be in the forward pipeline.

One machine is then doing two things at once: receiving or sending gradients for the previous batch, and computing the current batch. That is the single-machine overlap picture. Computation of the current batch and communication of the previous batch’s gradients share the same device. Both run in parallel. The processing time drops because the idle gap between “I finished math” and “I finished the network copy” shrinks.

A second numeric check uses unequal clocks. If compute is 10 ms and send is 2 ms, overlap buys you almost the whole 2 ms. If compute is 2 ms and send is 10 ms, overlap buys you the 2 ms of math and you are still network-bound. Overlap never invents bandwidth. It only refuses to add the two clocks when they can run together.

9.1.4 Ordered SGD versus Overlapped SGD

In ordered SGD, compute must finish first. Only then does communication start. That barrier is the problem. Overlap SGD cuts that latency. Quantized SGD, treated in the next section, attacks a different cost: the number of bits in each number. Overlap does not change how fat each gradient is. It changes when the send starts relative to the math. Communication volume can stay the same while wallclock time still falls, because the GPU stays busy instead of waiting on a barrier.

When to pick which: use overlap whenever the hardware can copy during compute and the send is large enough to be worth hiding. Use quantization when the size of the message is the bottleneck. Use both when you have a fat model on a slow link.

Gradient buckets are the systems trick that lets this schedule start early: you do not wait for the entire backward pass. The next section first shrinks the numbers themselves; buckets then pack those numbers so overlap has something to send.

On-device voice models and hospital imaging nets will later need the same idea: keep the accelerator busy while updates move. First, though, shrink the payload those updates carry.

9.2 Quantized SGD: Shrinking the Numbers You Move

Quantized SGD cuts the bit-width used to store and send values. A student restatement that matches the classroom target is: in quantized SGD we reduce the size of the data that is being transferred. Quantization here means representation, not a new optimizer. The algorithm that decides when to average or whether to wait for all workers can stay the same. What changes is how each scalar is stored: a wide float, a short integer, or even a single sign bit.

Quantization is not the same trick as overlap. Overlap cares about whether communication happens on one side of the machine while computation happens on the other. Quantized SGD “just works on the numbers.”

Everyday picture: you are mailing a box of receipts. Overlap is starting the next envelope while the truck is already rolling. Quantization is writing each amount as 8.3 instead of 8.321000 so the box itself is lighter. The analogy breaks when two nearby amounts round to the same short code: the truck is faster, but the bookkeeping is coarser.

Standard distributed-training notes say the same thing in memory language: use fewer physical bits to represent a single value. A 32-bit integer uses 4 bytes; a quantized code may use 2 bytes, 1 byte, or 1 bit. That transform is lossy. Values such as $1$ and $1.01$ can collapse to the same code, so you lose the $0.01$. Mixed-precision libraries (for example NVIDIA AMP wrapping an SGD optimizer) often keep some tensors in a 32-bit format and shrink others to FP16. The lecture’s ladder goes further, down through 8-bit and 1-bit, because communication volume is the live cost.

9.2.1 Bit-Width as the Control Knob

High-end representations include 32-bit floating point and 16-bit floating point. One spoken slip in class said “FP36”; the intended high-end type is FP32. Narrower choices include 8-bit integers and, at the extreme, 1 bit. The verbal list is: if you use a higher representation like FP32 and FP16, then 8, into 4, only 1 bit. (A muttered “98, into 4” is noise; later numbers lock onto FP32, 8-bit, and 1-bit.)

Those cuts drastically reduce communication time and can reduce computation time as well, because narrower types move faster and often run faster on the device.

Let $b$ be bits per scalar and let $n$ be the number of gradient values in one payload. Communication volume in bytes is

$$V(b)=n\cdot \frac{b}{8}$$

Relative to FP32,

$$\frac{V(b)}{V(32)}=\frac{b}{32}$$

So INT8 is $8/32=1/4$ of the FP32 volume, and a pure 1-bit code is $1/32$ of the FP32 volume before headers and padding.

9.2.2 Worked Size Example: 400 MB, 100 MB, and 12 MB

A demo starts from a gradient payload that needs 400 MB when each value is an FP32 float. One shown value looks like $0.8321$ in that wide format. After a step that switches to 8-bit integer representation (spoken as “B8”), the same payload is described as 25 percent of the original size, which is 100 MB. The classroom reason is the 8-versus-32 ratio: $8/32=1/4$.

Let $V_{32}$ be the communication volume at 32 bits per value. The verbal explanation is: if I use FP32 bit float, 400 MB is required for communication; if I do with 8-bit, I am able to do it with 25 percent; 100 MB is enough for the same communication.

That 1-bit scheme looks weird for ordinary neural training. It is not widely used. The practical default called out in class is 8-bit. The 1-bit story is for special settings: parity-style checks, outlier flags, or scientific codes where you already hold the original magnitudes locally and only need a plus-or-minus action from the other side. In those cases you send one bit and skip the bulky payload.

Error after quantization still exists. The demo shows a leftover error once gradients have been packed this way. That error is the price of the smaller envelope. You still complete the gradient step; you just complete it with coarser numbers.

Draw the ladder as a bar chart. Horizontal axis: representation (FP32, FP16, INT8, 1-bit). Vertical axis: payload megabytes. Landmarks: 400 MB at FP32, 100 MB at INT8, 12 MB at 1-bit. Takeaway: each cut in bit-width is a direct cut in bytes if you keep the same number of tensors.

9.2.3 Quantization on Compute, Communication, or Both

If you have ample compute and ample network, you can stay on a high-end type and skip quantization. If you must deploy on a low-end device such as a Raspberry Pi, you want lower representations and quicker communication, so you quantize. That is where the 25 percent and 3 percent cuts matter: they drastically reduce the representations you store and send.

A Raspberry Pi-class board has a small DRAM budget and a modest interconnect. An FP32 400 MB gradient buffer does not fit a comfortable training loop there. An INT8 100 MB buffer is already a different product story. The same 8-bit tensors also run as integer math on many edge chips, so both compute and communication get cheaper together.

9.2.4 Quantization Is Not a Separate Optimizer

The same correction is repeated later: quantized SGD is not a separate technique. You can use quantized values inside overlap SGD, inside asynchronous SGD, or inside synchronous SGD, because it is data only. It is not the algorithm. Communication size changes under quantization. Overlap timing changes under the overlap technique. The GPU-busy benefit of overlap is a scheduling effect. The bit-width benefit of quantization is a representation effect.

Real-world: 8-bit is the widely used practical point on this ladder. 1-bit sign-only traffic is reserved for niche scientific or parity-like needs. Raspberry Pi-class devices are the motivating low-end deployment.

9.3 Gradient Buckets: One Packet per Group of Gradients

Sending every scalar gradient as its own tiny message is slow. Frameworks bucket gradients: they group many gradients into one communication packet. The bucket idea is what lets overlap start early. You do not wait until the entire model’s backward pass is done. You wait until one bucket is full, then you ship that bucket.

The analogy breaks when the envelope is so large that you delay a neighbor who already needed the first receipts. Bucket size is that trade: too small and you pay stamps; too large and overlap starts late.

9.3.1 Why Individual Gradient Sends Hurt

Without buckets you might send on the order of a thousand messages, with a cited overhead on the order of five milliseconds in the demo story. That pair is a spoken classroom figure, not a derived law of physics. Each tiny send pays startup cost: queue a work request, ring doorbells on the NIC, wait for completion. The fix is: do not send continuously. Store gradients into a bucket, then send the bucket.

The verbal definition is: grouping the gradients; sending each gradient individually is very slow; frameworks group them into buckets and one communication packet per bucket. A size used in the demo is 500k gradients as one bucket, then that packet goes out to the GPUs so they can adjust their gradients together.

Let $B$ be the bucket length in number of gradient scalars (or in bytes, once you pick a dtype). A bucket is ready when the backward pass has produced $B$ entries. Then one packet leaves. Start sending as soon as the first bucket is ready. Do not send a smaller trickle if you have already chosen a bucket size. Decide the volume, fill it, then communicate back to the layers that need those gradients.

If the model has $P$ parameters, the number of full buckets is $\lfloor P/B\rfloor$, and the tail has $P\mathrm{mod}B$ leftovers. Overlap can start after the first bucket, which is after $B$ scalars exist, not after all $P$ scalars exist.

9.3.2 Buckets, Overlap, and the Tail

The compute–communication diagram with buckets looks like: compute, communicate, compute, communicate, and finally a tail. The tail is the leftover work after the last full bucket, including finishing the output-side communication. Because the first bucket can leave before later layers finish, communication rides under later compute. That is the link back to overlapping SGD.

Ordered SGD still waits: first the full compute, then communication. Overlap plus buckets cuts that wait. Quantization, again, is orthogonal: it shrinks each value inside the bucket.

Visualize two stacked timelines. Top lane: backward compute producing gradient blocks $G_1,G_2,\dots ,G_T$. Bottom lane: sends. In the bucket picture, send($G_1$) starts as soon as $G_1$ hits size $B$, while compute is already writing $G_2$. The landmark is the tail bar after the last full send: a short leftover copy that cannot hide under later math because the backward pass is done.

9.3.3 Student Questions: One Model, Bulk Updates, Not Every Layer at Once

Several students reached for a second, related confusion: maybe you must collect every layer, save, and only then update. That reading is plausible if you think of “a bucket” as “the whole backpack.” It is the wrong backpack.

So the misconception to drop is “collect the entire model, then send once.” The useful picture is: fill a bucket of chosen size, send that packet, keep computing the rest. Layer-1 gradients can leave before layer-$L$ gradients exist.

A 10k bulk update is a smaller classroom bucket than the 500k demo packet. Both numbers teach the same rule: pick a bulk, fill it, send it. They are not a contradiction. 500k is a systems-sized packet on the wire. 10k is the “do not update one or two parameters at a time” intuition.

9.4 Federated Learning: Train Locally, Share Updates

With the SGD communication variants done, the course leaves traditional distributed machine learning and moves to federated learning. FL is widely used when data must not leave its home. A first student definition is already in the right shape: whenever there is a constraint that data cannot be shared, you train locally and then share the gradients to a master, which aggregates them with a federated aggregator.

Textbook language matches that split. In ordinary data-parallel training, every worker can still see a global shard of a dataset the trainer owns. In federated learning, each user or node does not get global access to the whole training set. Workers keep local data private and communicate knowledge: local weights or local gradients. The local pile is often too small and too biased to train a useful net by itself, so you still need a server (or a peer mix) that combines those updates.

The kitchen picture from overlap still helps, with a new rule: the waiter is no longer allowed to carry the ingredients, only a note about how the recipe changed.

9.4.1 Device-Generated Data and the Privacy Constraint

Why refuse to share the data? Privacy rules and geographic regulations can block movement of records. The data is getting generated in the devices. That often means personal data. People do not want that raw stream sent to a company server. The FL move is: develop models on the device, share the models (parameters, weights, or gradients) with a server, and let the server build an aggregated version.

The verbal core is: I am sending this model instead of the data. When I say model, it can be the parameters, weights, or gradients. Parameters here means weights and gradients as the things you actually ship.

A four-step loop, aligned with the usual server–worker picture, is:

1. The server broadcasts the current global weights to the workers that are awake this round.
2. Each worker trains on its own local data only.
3. Each worker updates its local weights (many local SGD steps are allowed, because the link may be choppy).
4. The server collects those weights (or gradients) and writes a new global model.

Sharing a gradient after every mini-batch is often impossible: a phone may be offline. Sharing weights after a local stretch of training is the more practical message.

9.4.2 What Success Means: Match a Pooled-Data Model

Technically, many clients collaborate on one learning problem. One mental picture is: you could, in theory, gather every client’s samples onto one machine and train a single model on the union of those samples. FL’s requirement is that the aggregated model should behave as close as possible to that pooled-data model, even though the pooled matrix never exists in one place.

Let $D_k$ be client $k$’s local dataset and let $w^{\star }({\bigcup }_k{D}_k)$ be the model you would get by training on the union. Let $w_{\text{FL}}$ be the model after federated aggregation. The verbal requirement is: the single model which we are aggregating must be as close as possible to the model trained with all this data used in multiple clients. That is why aggregation is a problem, not a footnote.

$$w_{\text{FL}}\approx w^{\star }(D_1\cup D_2\cup \cdots \cup D_K)$$

If that match were automatic, FL would be a simple copy of centralized SGD. It is not automatic. Clients join and drop mid-round. Data is non-IID. Some clients send huge updates from tiny local sets. Aggregation has to respect those facts.

The $\approx$ is a design target, not an identity. Write a gap if you need a number:

$$\Vert w_{\text{FL}}-w^{\star }\left({\cup }_{k=1}^K{D}_k\right)\Vert$$

should stay small on the task metric you care about (loss, accuracy, word-error rate), not only in Euclidean weight space. Two weight vectors can look far apart and still classify the same, or look close and still fail on a silo the server never saw.

9.4.3 Roadmap: Types, Aggregation, and Later Papers

The rest of the course, as announced here, will study the fundamentals of federated learning (spoken “vertical” in one pass as a slip, then corrected into types), types of federated learning including horizontal and vertical learning, and various aggregation algorithms. Aggregation algorithms play a major role. How you pass the data, how differential privacy wraps it, and how you extract features all depend on that stack.

Supporting definitions that the lecture pointed at, and that reference notes spell out:

Horizontal FL is the Gboard-style, cross-device default. Vertical FL needs a private way to align identities (the notes mention private set intersection) and is the natural language for “same patient, different hospital departments.” Cross-silo FL is the hospital and bank setting: few fat institutions, not millions of phones.

Named methods already on the color-coded list include FedProx (spoken “FedProxy”) and SCAFFOLD. The plan is also to read later papers, including work from 2026, rather than only the older vanilla story.

FedProx keeps FedAvg’s averaging but changes local training. Slow clients may take fewer steps. A proximal term $\frac{\mu }{2}\Vert w-w^t{\Vert }_2^2$ penalizes wandering far from the current global weights $w^t$, which fights the client-drift that appears when local data is unlike the global mix.

SCAFFOLD (Stochastic Controlled Averaging) attacks the same drift with control variates: a server direction $c$ and a per-client direction $c_k$. The local step subtracts an estimate of how this client’s gradient disagrees with the global direction. You will swap the mixer, not the rest of the client loop, when those papers arrive.

9.5 Data Impressions: When Shared Weights Leak Training Patterns

The vanilla FL pitch is: I will not share data; I will share the model. A research paper discussed in class attacks that pitch. Pre-trained deep models hold latent knowledge in the form of model parameters. Those parameters act as memory for trained models and help them generalize on unseen data. If you only have the trained model and not the training set, people used to say the model is still useful for inference or as a better initialization for a new task. This paper goes further: it extracts synthetic data by using the learned model parameters. The authors dub those samples data impressions. Data impressions act as a proxy to the training data and can be used for a variety of tasks.

The classroom target is not a new generator network trained on extra public photos. It is: take a trained network’s parameters and mine samples that behave like the original training distribution. The published method names those samples data impressions because they are the training set as the model understood it.

9.5.1 Student Readings of the Paper, Then the Correction

A second, sharper mix-up compared the paper to autoencoders.

Another restatement that was accepted: large models have learned from a huge amount of data; for a specific task you may lack data; you can try to create data from that large model. The paper’s own highlight, read in class, is the sentence about extracting synthetic data by using the learned model parameters, and dubbing them data impressions that act as a proxy to the training data.

9.5.2 Last-Layer Templates: Cat, Dog, Elephant

Why would weights contain the training distribution? Because a classifier learns a template per class. Look at the last layer of a network and the layer just before it. One output node is the cat probability, one is the dog probability, one is the elephant probability. The weights that feed the cat node are the features for the cat class. The weights that feed the dog node (drawn green in the demo) are the dog features. Another bundle of weights is some other class. If you can extract the last layer, or one step before it, you learn which classes exist and what feature directions those classes used.

At test time, an input is pushed through the net until it becomes a feature vector. That vector is compared, in effect, to the class templates. If the extracted features of a test sample sit closer to the cat template than to the dog template, the model calls it cat. Those templates were carved from real training images. Sampling around them (the classroom phrase is “digital sampling, some sampling techniques”) can yield impressions that look like the original trained data.

The verbal logic of the attack is: every model learns a template for every class; the previous layer’s connections into this node are the features learned for this class; each node learned the template for each class; it is a vector; how close a test sample’s features are to that vector decides the class. If you hold the templates, you hold a generative handle on the training patterns.

The paper’s extra algebra, after that template picture, is a way to pick the target softmax vectors. From the last-layer weight matrix you read a class-similarity structure: classes that share features have aligned template vectors. Those similarities become concentration parameters of a Dirichlet distribution over $K$-class probability vectors. You sample a soft label from that Dirichlet, then optimize the input until the teacher network’s softmax matches the sample. No training images and no extra metadata are required. The lecture’s cat–dog–elephant picture is the same last-layer handle, told without Dirichlet notation.

9.5.3 The FL Principle Breaks, Then Differential Privacy Enters

If a server (or a thief who intercepts updates) can mine impressions from the shared parameters, the fundamental principle on which federated learning is built is gone: “I did not share data” is no longer a full privacy guarantee. Similar samples can be generated from the model parameters you just sent. The fundamental principle of federated learning fails if impressions reconstruct training data.

The mitigation named here is differential privacy (DP), plus an encryption layer. Rather than sending the model in the clear, you apply private techniques, add encryption, and only then send. On arrival you decrypt (in the allowed protocol) and continue training. Classroom pointers also include a privacy-in-AI thread and the monograph title The Algorithmic Foundations of Differential Privacy. Secure aggregation is named alongside DP: carefully calibrated random noise is added so that the contribution of one individual becomes difficult to identify, and you may also encrypt the information.

Reference notes on DP match that recipe. Each participant can add random noise to local parameters, often from a zero-mean Gaussian or Laplace law, so that one person’s influence is hard to spot. When many parties aggregate, those zero-mean noises tend to cancel, and the global model stays close to the no-noise mix. The catch: with only a handful of hospitals the noise may not cancel, and the global model suffers. Secure aggregation and (partial) homomorphic encryption are the complementary tools: mix under encryption so the server sees a sum, not each raw update.

A later analogy uses signatures. Suppose you have English signatures and a model trained from millions of signatures. An attacker takes the shared model, mines impressions, reframes data from those impressions, and trains a new task that had no labels of its own. The stolen proxy set becomes that attacker’s training set.

Real-world: this is the break that turns FL from “just average the weights” into a privacy-engineering problem.

9.6 Voice Assistant Workflow: Central Collection versus On-Device Training

A running demo uses a voice assistant in the style of Siri on iPhone hardware, with Apple as the central organization. The product goal is that the assistant should self-learn to recognize the user’s voice more accurately. The same workflow is the teaching scaffold for FL: traditional centralized machine learning first, then the federated reverse of data movement.

Think of tutors visiting each home instead of every household mailing its diary to one school. The tutor (the model) travels. The diary (raw voice) stays. The analogy breaks when the tutor’s notebook still describes the diary well enough for a stranger to fake new pages — that is the impressions attack from the previous section, and it returns at the end of this one.

9.6.1 Traditional Centralized Voice Training

In the old centralized system, the company could collect data. When you speak, the phone connects to the company server. Actual audio samples spoken by users are collected: commands such as “call mom” and “open maps.” Regional and individual speaking styles are captured, including how words are spoken, and sometimes background noise. The problem: private data leaves the user’s phone. Even if the link is wrapped, the central organization may still need to store or process very large quantities of personal data. Personal voice clips and commands leave the device. A central database holds them. Administrators of that store, if they know the password or the key, could listen to all the voice. Systems must also comply with privacy laws and data protection rules. If the server is compromised, the leak is the raw audio.

The design question FL is built to answer: can the assistant improve without collecting and centrally storing every user’s raw audio?

9.6.2 Reverse the Data Movement

Federated learning moves the training process closer to where the data already exists. Training happens at the data location: the user device. iPhone A keeps using its voice data locally. In a picture that still shows “voice data uploading,” the old world uploads millions of voice samples, trains on the server, and sends a better assistant down. FL reverses the usual direction of data movement. Instead of moving raw voice data, the company creates an initial assistant model and sends that model to selected phones. Each phone trains using its own data. Only model updates are returned. Raw voice stays on the phone. The phone sends only the local learning: parameters of what it learned; only mathematical model updates.

Steps in order:

1. The company prepares model version 1, a vector of learned parameters (weights). For teaching, pretend there is a single weight. In reality a neural net has many numerical weights.
2. That same initial weight is sent to a selected group of phones. Example initial value: 0.42.
3. Local commands differ. One user says “call mom.” Another says “open maps.” Clips remain on that user’s phone and are used only for local computation.
4. After local training, the weight moves. Different users speak differently and sit in different environments, so each device learns a slightly different update.
5. Phone A sends an update of +0.05 (one feature). If there were 100 features, there would be 100 such numbers. Other phones send their own updates.
6. The server aggregates (average or weighted average). It builds model version 2 and redistributes it. Meanwhile new sensor-like data keeps appearing (the demo also mentions temperature and similar on-device streams). Devices train again on the latest model. The loop repeats.

The verbal explanation of the reverse is: traditional machine learning moves raw voice data toward the server; here we create an initial model, send it to selected phones, each phone trains using its own data, only model updates are returned, raw voice data stays on the phone.

9.6.3 Worked Numbers: 0.42, Plus 0.05, Average, Then Version 2

For easy understanding the model is treated as one weight. Everyone starts at $w=0.42$. Phone A, after local training, reports an update ${\Delta }_A=+0.05$. The local weight on that phone is

$$w_A=0.42+0.05=0.47$$

The verbal explanation is: old weight is 0.42; phone A will send update 0.05; 0.47 I updated. Other phones send their own signed updates (plus or minus). The server’s job in the vanilla loop is to average those updates (or the resulting weights) and push version 2 to everyone.

If $K$ phones send updates ${\Delta }_1,\dots ,{\Delta }_K$ and we take a plain mean:

$$\bar{\Delta }=\frac{1}{K}\sum _{k=1}^K{\Delta }_k,w^{(2)}=w^{(1)}+\bar{\Delta }$$

Class also described averaging the local weights themselves. Both views appear in the FedAvg section. They match when every client starts from the same broadcast $w^{(1)}$ and reports a full new $w_k=w^{(1)}+{\Delta }_k$.

Millions of users can participate over time. Every phone contributes experience from a different user environment: American English, British English, older speakers, car noise, office noise. Apple need not receive raw information: no plain audio, clips, conversations, or commands are shared. What is shared are model parameters, gradients, weight differences, and counts such as how many local samples were used.

Personal voice now enters the local model, gets trained there, and the gradients or weight tables are what travel to the server.

9.6.4 Traditional ML versus Federated Learning, Side by Side

Traditional machine learning: data goes to the server; that creates stronger centralized privacy concerns; it requires a large central store of user data.

Federated learning: raw data stays on the device; in-device data never takes that trip; there is no need to store central raw audio; in its place you send gradients that still must be stored and aggregated. FL also introduces challenges: unreliable devices, communication limits, and hard learning dynamics. Those are not small print. They are the next technical chapters.

When to pick which: if you legally own a single IID pile and a cluster, classical distributed training is simpler. If the data cannot leave the phone, FL is the setting, not a synonym for All-Reduce on your GPUs.

Real-world: Siri-style on-device voice improvement is the product story. The same loop applies anywhere a phone or sensor owns private audio, text, or telemetry.

9.7 Federated Averaging: Plain Mean versus Sample-Count Weights

Federated averaging (FedAvg) is the basic aggregation rule. Every client reports local weights (or gradients). The server mixes them into one global weight and sends that mix back as the next model version. You may send weights or gradients; the server holds those values and updates.

Textbook FedAvg is the parameter-wise mix of local models $M_1,\dots ,M_K$ with local dataset sizes $n_1,\dots ,n_K$ and $N={\sum }_k{n}_k$:

$$M_{\text{agg}}=\sum _{k=1}^K\frac{n_k}{N}{M}_k$$

The original algorithm also samples a fraction $C$ of clients each round rather than waiting for everyone. Class here focuses on the mix itself: plain mean versus $n_k$-weighted mean. Later papers replace that mixer (FedProx, SCAFFOLD) without rewriting the rest of the loop.

9.7.1 Vanilla Average

In the vanilla version, whatever you send, the server takes the average. If phone A sends $0.05$ as its update and others send theirs, the server averages. Then it redistributes model version 2. The verbal picture is: every client will respond “these are all my local weights,” and this is the aggregate weight after doing this; suppose 0.47 I updated; compute the average value and then send back to all other users.

For $K$ clients with updates ${\Delta }_k$,

$${\Delta }_{\text{plain}}=\frac{1}{K}\sum _{k=1}^K{\Delta }_k$$

Every client has mass $1/K$, including a client who saw one sample.

9.7.2 Weighted Average by Local Sample Count

A weighted version also sends $n_k$: how many local samples phone $k$ used. Phone A used $n_1$ samples to justify its $0.05$ recommendation. The server forms a weighted average of updates (or of weights):

$${\Delta }_{\text{global}}=\frac{n_1{\Delta }_1+n_2{\Delta }_2+\cdots +n_K{\Delta }_K}{n_1+n_2+\cdots +n_K}$$

The verbal explanation is: $n_1$ into the weight you said 0.05 plus $n_2$, and totally we will take the average; weight average of the updates. You can set how the global weight will be; you can play with that mix. This is also a live research area: metrics for building a global model are not a blind average.

The same $n_k$ weights are the first bias knob. A client who trained on one sample should not drown a client who trained on a hundred.

9.7.3 Bias, Accents, and Why $n_k$ Is Not Enough

A second bias is linguistic, not fake labels.

Sample-count weighting shrinks a one-sample client so it cannot overflow the global model. A single model with high bias, high loss, and high gradients should not overflow the global model creation.

Advanced methods go further than $n_k$. Those are later in the course. The sample-count weight is the simple technique shown here.

9.8 Cross-Silo Federated Learning for Hospitals

Map the voice-assistant loop onto medicine. Leave “privacy” as a slogan and name a scenario.

Different hospitals, different patients, different sensor nodes, and different (non-IID) data: that is a natural FL setting. Hospitals often will not share raw records. That is where this comes into the picture. Healthcare is also where a lot of early FL product energy went: medical imaging, drug discovery, and electronic health records all hit the same wall of data that cannot legally sit in one vendor cloud.

9.8.1 Mammogram Demo across Sites

A cross-silo demo: a federated server stores the global model. Hospital A captured local mammograms specific to its area (an India-area site in the story). Another hospital is in the US. Others are in other countries. Data stays with the hospital. You are not going to leak images, names, reports, history, or doctor annotations; those remain inside the hospital. For this teaching pass, assume the counter-paper (data impressions) is paused and that sending gradients is “safe,” so we can see the statistical gain of collaboration.

The loop matches Siri:

1. The server creates a global model and sends the same one to all hospitals.
2. Local training uses local mammograms.
3. Each hospital returns model changes (weight updates, gradients, or other aggregate information, possibly wrapped for transit).
4. The server aggregates and redistributes. Another round begins.

Hospital A’s model learns dense breast tissue patterns. Another hospital learns a different set of patterns from a different scanner: rare abnormalities, early-stage patterns, other tissue patterns. When you combine those experiences, quality can rise. The final global model may generalize better across hospitals, patient groups, and imaging equipment, because it saw images from different devices, not only Hospital A’s scanner. How you combine that information is the scientific core.

9.8.2 Worked Aggregation: 10,000 Samples Times 0.6

Hospital A has 10,000 local samples and a local trained weight (the demo pairs this with a factor 0.6 in the weighted product). You can take a simple average: blindly mix the upgrades sent by various hospitals. Or you take a weighted average because 10,000 samples should count more than a tiny site:

$$n_A=10,000,n_A\times 0.6=6,000$$

The global model is then redistributed and another training round begins.

Real-world: mammography across countries and scanners is the medical story. The same silo pattern applies to any hospital network that will share parameters but not DICOM dumps.

9.8.3 Why the Medical Model Gets Better

Each hospital contributes knowledge from its own environment. Sample counts grow in effect because experience is pooled through parameters. More important than raw count: different learning patterns, different devices, different patient groups (one site may see older patients, another middle age). Equipment differences enter the high-level model. In some cases that is exactly why federated learning succeeds.

Challenges still bite. Non-IID data: each hospital may have a different patient population and disease distribution. That is the main problem. A harder variant: one hospital stores cardiology information, another eye information, another kidney information. Feature distributions then differ. Training one model across those distributions is more challenging. Weight aggregation, domain adaptation, and strong combination all matter. Compression is mentioned as not having much impact in this demo.

Cardiology versus eye versus kidney is already a vertical-flavored split: different feature families, sometimes different patients, often both. Horizontal mammography (same image task, different people and scanners) is the easier collaboration. Mixing heart traces with retinal photos into one net is the harder one, and it is why aggregation research is not optional.

9.9 Challenges, Domain Adaptation, and Federated versus Central Distributed Training

Federated learning is a shift from a centralized data pile to training on locked-in machines. Data is locked in the devices. Models move to the data, train, and recombine into a global model. Step one: send the current global model out. Step two: send updated models or gradients back and aggregate. Privacy issues, communication cost, many clients, missed rounds, new clients joining later, data heterogeneity, system heterogeneity, mixed network bandwidths, mixed hardware, and security all make FL hard.

9.9.1 Why Federate at All

Sending the data may be too costly compared with sending the parameters. Self-driving cars are expected to generate several terabytes of data a day. Some wireless devices have limited bandwidth, so you cannot ship the raw stream; you can ship model parameters. You may also be blocked by sensitivity: you cannot send data, but you can send parameters. A local dataset may be too small and may overload a purely local model; you need data from multiple clients. Bias in one silo can be reduced by adding other clients, using the $n_k$ idea and later algorithms.

In distributed training in the classical lab sense, data is centrally stored in a single place, often treated as IID, and you do not pay the same “move hospital records across borders” threat. In federated learning, data is naturally distributed. There is no single central place to store it. Data is not independent and is not identically distributed. Parameters collected in one hospital may follow a different distribution than those from another hospital, and you still have to combine them. Class imbalance across silos is the same story: maybe only two patients went through cardiology, while 100 people went through the eye clinic.

Coordination can be a single server, or peers can talk among themselves to come up with a common model. That peer mix is the decentralized picture: no one silo has to own the hub.

When to pick which: if you already have the pile and the cluster, use classical data-parallel SGD (with overlap, quantization, and buckets as needed). If you cannot move the pile, you are in FL, and FedAvg is the first mixer, not All-Reduce on shared disk.

T1’s edge-device list matches the classroom pain: weaker chips than data-center GPUs, battery budgets, flaky links, devices that power off, users who produce noisy labels. Those are system heterogeneity, not footnotes.

9.9.2 Domain Adaptation as a Related Idea

The verbal summary is: related domain, get those samples, use that information for a new domain; the model is trained with one dataset and you want to test with a different dataset; that is adapting.

In the hospital FL setting, domain adaptation sits next to aggregation: scanners, populations, and disease mixes are different domains that the global model must survive.

Kannada character recognition can borrow Telugu and Tamil scripts at about 70 percent. That is the same 70-not-90 lesson as the signature story, told with South Indian scripts.

9.9.3 Student Summary of the FL Loop

Real-world: IoT sensor data, medical data, in-device phone data, and self-driving terabyte-per-day logs are the motivating sources.

9.10 Feature Completion from Overlapping Patients

Privacy and bandwidth are not the only reasons to federate. A feature-level bonus appears when the same entity shows up in several silos with different measured fields.

This is the vertical flavor from the FL types roadmap: same sample IDs, different columns. Horizontal FL (same columns, different people) does not by itself fill a missing column. Overlap patients are the bridge.

9.10.1 The Overlap Picture

Think of three hospitals and three people, A, B, and C. Person A is present in all of them: a cardiac hospital, another site, and a third site (the demo names a cardiac hospital in the style of a large private chain, then an X hospital and a Y hospital). Cardiac features, heart features, and eye features are not stored together. Person B is in the cardiac silo along with A, but some of B’s columns are missing there and present elsewhere. Hospital 1 has a subset of columns. Hospital 2 has a different subset. Hospital 3 has yet another. For the same person B, feature block 1 is missing in one silo, feature block 2 is missing in another, and feature block 3 is missing in the third. Federated collaboration can help fill the holes. Shipping huge raw tables is a different story (bandwidth). The extra FL-shaped prize is this column completion.

The verbal claim is: this federated learning will help me to fill these features; I can get these features; that is the advantage, more than only shipping smaller messages.

How? Learn the correlation on overlapping patients, then apply it where a column is missing. A toy numeric pattern:

• In one silo the overlapping pair looks like $1,2,3$.
• In the other silo the matching overlap looks like $2,4,6$.
• That suggests a $\times 2$ relation between the two views of A and B.

Carry the same pattern to a row that is only partly known: something like $1,4,?$ filled as $6$ in the toy, and $2,8$ filled as $12$. The classroom warning is that real filling is not this simple; the toy only shows “learn the overlap pattern, then apply it where values are missing.”

These shared people are overlapped patients: present in more than one hospital. If a few features are missing, use the places where features are present to learn the pattern, then fill.

9.10.2 Do Not Drop Incomplete Records

If 80 percent of features are present and 20 percent are missing, you should not throw out the patient. You have to fill. A numbered filling path:

1. Find overlapped patients: the same person appears in more than one hospital.
2. On those overlapping rows, estimate a relation among the columns that are present on both sides (the $\times 2$ toy, or a richer fit in real data).
3. For a person who is missing 20 percent of fields, apply that relation to the missing slots.
4. Feed the completed row to the model. Do not drop the row only because a minority of fields is empty.

Overlap is that filling path: for A you have all the samples; for B you are missing some; understand the overlapping pattern; apply it wherever those features are missing. Once the features are filled, you automatically feed that vector to the model. The missingness itself is no longer a reason to discard medical history.

Do not throw out a patient when 80 percent of features are present and 20 percent missing. That warning is the clinical version of “missingness is not a delete key.”

9.11 FedAvg in Code: Update Size and Client Fractions

A code walk-through after the break shows local training, a size measure of the update, and a fed_average routine. The main engineering claim: aggregation is the hard part. FedAvg is written as its own function so later papers can replace that mix without rewriting the rest of the loop.

9.11.1 L2 Norm of the Weight Delta

After local steps you have new weights. You also compute a scalar size of the update: the L2 norm of the difference (spoken “L2 norm of delta”). That number measures the size of an update. The verbal recipe is: for $W$ in the weight list, take a square-root / sum form that is the L2 norm, and share that to the server as well as appending the new weights.

The speech mixed “square root of W” with “sum of squares.” Use the usual Euclidean norm unless the code shows otherwise. For a list of tensors, flatten the delta into one vector (or sum the squared entries across tensors) and take the square root.

Walk the client loop as the demo code does:

1. Train on the local set and get new weights.
2. Form the per-tensor difference $\Delta w=w_{\text{new}}-w_{\text{old}}$ (the delta).
3. Reduce that delta to $\Vert \Delta {\Vert }_2$ and keep it as a size report.
4. Append every new weight tensor into the payload you will upload.
5. Send the new weights (and the size) to the server.

The server is not a blind dump. It runs FedAvg using client sizes (how many local samples each client used). It loops over the clients it heard from, applies each weight delta into a running mix, and then writes a new global model. Global weights are copied into that global model. After the mix, the demo evaluates: it reports loss and accuracy so you can see whether the round helped. That evaluate-after-aggregate step is how the demo shows that the model learns. The hard piece remains the mix, not the local SGD.

The standard reconstruction of that spoken L2 is the Euclidean norm:

$$\Vert \Delta {\Vert }_2=\sqrt{\sum _{w\in \mathcal{W}}(\Delta w{)}^2}$$

If the weights are a collection of tensors, the sum runs over every scalar entry of every tensor. In vector form, $\Delta \in {\mathbb{R}}^P$ for $P$ parameters and $\Vert \Delta {\Vert }_2=\sqrt{{\Delta }^{\top }\Delta }$.

Clients send new weights (and sizes). The server loops over clients, applies weight deltas, builds a new global model, writes global weights into that model, evaluates loss and accuracy, and repeats. That is how the model learns in this demo. The challenging task is still how you aggregate.

Time cost: each round ships $O(P)$ numbers per participating client. Space on the server is one running mix of size $P$, plus whatever client payloads you buffer. It gets slow when $P$ is huge and $K$ is huge; that is why quantization and client sampling ($C<1$) return as systems knobs.

9.11.2 Layer-Wise Client-Size Fractions

The separate fed_average function takes the client weights and the client sizes (how many samples each client used). It computes a total, then a fraction per client. For each layer, it multiplies that client’s fraction by that client’s weights in that layer, appends, and averages across clients. For that layer, it takes the average over all clients, weighted by those fractions.

Let $n_k$ be client $k$’s sample count, $N={\sum }_{j=1}^K{n}_j$, and $w_{\ell }^{(k)}$ the weights of layer $\ell$ on client $k$. The mix is

$$w_{\ell }^{\text{new}}=\sum _{k=1}^K\frac{n_k}{N}{w}_{\ell }^{(k)}$$

The verbal explanation is: it got the clients; taking the total; client size average; fractions it computed; with the fraction of that client it is multiplying with that client weight in that layer; taking the average of each, all the clients’ average; for that layer it is taking the average for each layer.

That is the same $n_k$ idea as in the Siri and hospital demos, now written per layer so convolutional layers and dense layers each get a consistent mix. A conv kernel and a dense matrix are different shapes, but each is a tensor you mix independently with the same scalar fraction $n_k/N$. You do not average a conv tensor with a dense tensor. You average conv with conv, dense with dense.

Real-world: this is the skeleton you replace when a paper proposes a smarter aggregate. The rest of the client loop (local SGD, send, receive) stays.

Exam Guidance Summary

This session does not hand out a mark table. It does lock a study map for the federated half of the course and a contrast you should not blur on an exam.

1. SGD communication variants: Overlap SGD vs quantized SGD vs buckets. Overlap changes timing of compute vs send. Quantization changes bit-width (FP32 400 MB → INT8 100 MB → 1-bit 12 MB in the demo). Buckets pack ~500k gradients (or a 10k bulk update) so you do not send one scalar at a time. Quantization is representation, not a new optimizer, and it composes with sync, async, and overlap.
2. Federated learning definition: Train at the data location. Share weights or gradients, not raw records. Success means the aggregate is close to a model trained on the union of client datasets.
3. Attack and defense: Data impressions mine synthetic samples from shared parameters via last-layer class templates. Differential privacy, secure aggregation, and encryption are the named fixes. Do not claim FL alone makes updates non-informative.
4. FedAvg arithmetic: Plain mean vs $n_k$-weighted mean. Toy values 0.42, +0.05, 0.47. Hospital product $10,000\times 0.6$. Per-layer fraction $n_k/N$. L2 norm of the delta as an update-size statistic.
5. Challenges: non-IID silos, systems heterogeneity, client join/leave, small and biased local sets, bad or instrumented data, accent or population bias, domain shift (Kannada from Telugu/Tamil, SVC vs the 800-sample signature set, Japanese signatures toward English verification at ~70% not 90%).
6. Extra FL value: overlap patients and feature filling when 20% of fields are missing; do not drop the row.
7. Named follow-ons: FedProx (FedProxy), SCAFFOLD, 2026-era papers, horizontal vs vertical FL, The Algorithmic Foundations of Differential Privacy. Read the shared code before the next meeting; the next meeting goes deeper on these algorithms.

Key Industry Applications

1. On-device voice assistants (Siri / iPhone / Apple): Traditional path uploads “call mom” / “open maps” audio, styles, and noise to a central store. Federated path sends model version 1 (example weight 0.42) to phones, trains on-device, returns updates such as +0.05, averages, and ships version 2. Raw audio never needs to enter the server.
2. Low-end and edge hardware: Raspberry Pi and other tight devices use 8-bit (or narrower) tensors so compute and communication fit the box. 1-bit sign traffic is for niche scientific or parity-like needs, not the default product path.
3. Hospital silos and mammography: India-area, US, and other sites train on local mammograms (dense tissue, rare findings, different scanners). Names, images, reports, and annotations stay inside the hospital. Parameters move. Sample counts such as 10,000 reweight the mix. The same pattern shows up in medical imaging more broadly: large daily scan volume, strict privacy rules, and scanners that do not match across sites.
4. Self-driving and wireless IoT: Cars can generate several terabytes per day. Radios cannot always ship that volume. Parameters are the affordable message. IoT and in-device medical sensors are the same pattern.
5. Signature and script systems: English / Japanese / French signature verification and Telugu / Tamil / Kannada character recognition show domain adaptation: related domains transfer some accuracy (about 70%, not 90%) when the target script or language has no samples. SVC versus an 800-sample signature set is the dataset-shift cousin of that story.
6. Privacy stack: Data-impression mining against shared deep models; differential privacy noise; secure aggregation; encryption of updates; literature pointer The Algorithmic Foundations of Differential Privacy; aggregation research FedProx and SCAFFOLD.
7. Peer FL: A single federated server is the usual picture; peers can also mix among themselves toward a common model when no one silo should own the hub.

These deployments share one engineering spine: keep raw records at the source, ship updates, mix with $n_k$ when sample counts differ, and do not treat those updates as empty of personal detail.