Federated Averaging and SCAFFOLD

10.1 Local Data, Shared Models, and Vertical Feature Completion

Federated learning is a way to train one shared model across many devices or institutions without ever pooling their private records in one place. The picture that opens this unit is industrial, not abstract. Phones train a shared next-word predictor without uploading keystrokes. Hospitals train a shared diagnostic model without shipping patient files. Banks train a shared risk model without sending account ledgers.

The rule is simple and it does not change later in this material. Data stays on the device or in the local system. You never share the raw data. You always share the model, which means you share parameters, not examples.

Each site already has data. Training still needs a useful model. So each site trains locally on what it already has. Then it sends an updated model, not the dataset, to a central server. The server combines those models and sends a new shared model back.

Think of a neighborhood cookbook. Every household keeps its own recipes in a locked kitchen. Nobody photocopies the family notebook. What travels is a short list of “tweaks to the shared batter recipe.” The village baker mixes those tweaks into one batter card and sends the card back. The mapping is: kitchen = local disk, recipe notebook = private dataset, batter-card tweaks = model parameters, village baker = server. The analogy breaks when two kitchens cook entirely different cuisines: averaging batter tweaks then yields a compromise that fits nobody, which is the non-IID problem taken up in Section 10.5.

10.1.1 The Core Rule: Keep Data Local, Share Only the Model

A client is one participating device or organization. A server is the coordinator that holds the current shared model and runs the merge step. The shared model is a vector of numbers called parameters. We write that vector as $\theta$. Each client $k$ holds a private local dataset $D_k$ and never transmits $D_k$.

In a linear model, $P$ matches the feature length (plus a possible intercept stored as one extra weight). In a deep net, $P$ is the full count of weights and biases. The same contract covers both.

The first practical algorithm for that contract is Federated Averaging, often written FedAvg. It is the vanilla baseline: each chosen client runs local training, then the server takes a sample-size-weighted average of the returned parameter vectors. Later we study SCAFFOLD. The expansion used in this unit is Stochastic Control Averaging for Federated Learning. The standard paper title is Stochastic Controlled Averaging for Federated Learning. The extra “-ed” is a naming detail, not a new method. SCAFFOLD is a repair for a failure mode of FedAvg, not a replacement of the privacy rule.

Standard form of the same contract, written as a one-round loop:

$$\begin{array}{rl}\theta & \leftarrow \text{current shared parameters},\\ {\theta }_k& \leftarrow \text{ClientUpdate}(\theta ,D_k)\text{for each chosen client }k,\\ \theta & \leftarrow \text{Aggregate}(\{{\theta }_k\}).\end{array}$$

FedAvg fills in a specific ClientUpdate (local mini-batch steps) and a specific Aggregate (weights $N_k/N$). SCAFFOLD fills in a corrected ClientUpdate and also ships a small extra vector called a control variate. Both still send models, never $D_k$.

10.1.2 Vertical Feature Completion Across Sites

Before the FedAvg derivation, the discussion finished a related industrial pattern: vertical federated learning. In vertical federated learning, different sites hold different feature groups for overlapping people, not just different people with the same feature list.

A feature is one measured field, such as a lab value, a skin finding, or an eye-scan score. One hospital may see the eye fields. Another may see cardiology fields. A third may see skin fields. The same patient can appear in more than one site, but no site has the full row.

Horizontal federated learning, which occupies the rest of this material, is the other axis: many clients hold different rows (different people or devices) with a similar feature layout, and they share models rather than rows. Reference treatments call horizontal FL sample-based and vertical FL feature-based. A bank and an e-commerce firm may both know the same city residents, but the bank holds credit fields and the shop holds purchase fields. They can train together without sending the joined table.

The practical job in the vertical picture taught here is often to fill missing features. If site A lacks fields that site B has, we look at people who appear in both views, measure how their overlapping fields line up, and then infer the missing fields for people who appear in only one view. More advanced completion methods exist, including encrypted matching of shared identifiers. The classroom picture is the simple neighbor-style comparison: who is close to whom on the shared fields, then copy or blend the missing fields from the close neighbor.

Picture three spreadsheets for the same people, each missing a different block of columns. The overlapping columns are the bridge. Distance on the bridge tells you whose missing block is a reasonable fill-in. The analogy breaks if the overlapping columns are weak: two patients can look alike on skin scores and still differ on eye fields.

When to pick which: use horizontal FedAvg when each phone, hospital, or bank holds its own pile of similar-shaped rows. Use vertical completion when one person is split across departments or companies and the missing fields are the bottleneck.

10.1.3 Hospital Walkthrough: Patients P1 and P2

A tiny numeric sketch of the same neighbor idea, using made-up overlapping scores so the comparison is visible:

• Shared cardiology score: $P_1=2.0$, $P_2=2.1$.
• Shared skin score: $P_1=4.0$, $P_2=3.9$.
• Distance on the shared plane: $\sqrt{(2.1-2.0{)}^2+(3.9-4.0{)}^2}=\sqrt{0.01+0.01}\approx 0.14$.

If another complete patient $P_3$ sits at cardiology $5.0$ and skin $1.0$, the same distance from $P_2$ is much larger. The fill for $P_2$’s missing eye fields should come from $P_1$, not from $P_3$.

Visual intuition: draw a table with people as rows and departments as column-blocks. Shade the eye block empty for $P_2$. Draw arrows from the cardiology and skin blocks of $P_1$ and $P_2$ into a “closeness” check, then an arrow into the empty eye block. The takeaway is that the empty block is filled from a neighbor on the shared blocks, while raw rows still never leave their site.

10.1.4 Industry Applications

Vertical completion is used when a patient, customer, or user is split across departments or companies. Horizontal FedAvg is used when each phone, hospital, or bank is a client with its own local sample pile. Both keep raw records local. Both send models or derived parameters, not the private table.

A bank and a retailer that share customers but not columns are the classic vertical pair: they can join feature types without shipping the joined ledger. Cross-silo hospital networks are the classic horizontal pair: each hospital is a client with its own patients and a similar chart layout.

In the broader field, this is how on-device keyboard models, multi-hospital diagnostics, and multi-bank fraud models get trained under privacy law: knowledge moves as parameters, not as records.

10.2 Clients, Local Datasets, and the Federated Optimization Goal

FedAvg starts from a concrete inventory. There are $K$ clients. Client $k$ has a local dataset $D_k$ with $N_k$ points. Clients need not have the same number of points. $D_1$ may be small. $D_2$ may be huge. That size gap will matter as soon as we average.

If we could pile every local set into one table, the combined dataset would be the union of the client sets. The total sample count would be the sum of the client counts. Federated learning tries to reach a model that is good on that combined table without building the table.

Think of $K$ locked filing cabinets. Each cabinet holds its own folder count $N_k$. The goal is not “make cabinet 1 happy.” The goal is a filing rule that would work if you could read every folder. You never dump the folders onto one desk. You only pass around the current rule card $\theta$.

10.2.1 Setup: $K$ Clients and Uneven Local Data

Let $K$ be the number of clients. For each client index $k\in \{1,2,\dots ,K\}$:

• $D_k$ is the local dataset of client $k$.
• $N_k=|D_k|$ is the number of local points on client $k$. $N_k$ is a positive integer. It can differ across $k$.

The spoken inventory was: there are $K$ clients; each client $k$ has a local dataset; $D_1$ may have $N_1$ points; $D_2$ may have $N_2$ points; each may have different data points.

A data point here is a pair $(x_d,y_d)$. The vector $x_d$ is the feature vector of example $d$. The value $y_d$ is its label. Every client stores its own pairs. No client sends those pairs to the server.

10.2.2 The Combined Dataset and Total Sample Count

The combined dataset $D$ is the union of the local sets. If you combine all these data, the overall dataset equals adding all the datasets. If you perform the union operation, you get the overall dataset.

$$D=\bigcup _{k=1}^K{D}_k$$

The total number of entities, written with a sum, is

$$N=\sum _{k=1}^K{N}_k$$

Here $N$ is the total sample count across clients. $N_k$ is the count on client $k$ only. Bounds run over clients $k=1$ to $K$.

If some people appear in more than one client, a strict set-union could count them once while a sum of sizes would count them twice. The rest of the derivation treats $N$ as the sum of local sizes. That is the usual FedAvg accounting when clients hold disjoint rows (a partition of the combined table). Standard write-ups define $N=n_1+n_2+\cdots +n_K$ the same way, with $n_k$ for local size.

Spot-check. Two clients, $N_1=3$, $N_2=7$, disjoint rows. Then $N=10$ and $|D|=10$. If the same three people also sat in client 2, a set-union would have $|D|<10$, but FedAvg still uses $N=3+7=10$ as the weight denominator. This unit follows the disjoint-row convention.

10.2.3 What We Actually Optimize

The goal is not “move weights with the least network lag.” The goal is a single parameter vector $\theta$ that makes loss small on the combined data. People think about eliminating loss across the complete dataset, not only on $D_1$, $D_2$, or $D_3$. We take the union of the datasets and try to minimize loss with respect to that union. The dataset $D$ contains the features, and the objective of the global loss is to minimize with respect to the global dataset.

Let $\theta$ be the global model parameters. The picture treated $\theta$ as a vector in ${\mathbb{R}}^P$, with a tiny running sketch of a three-dimensional vector when $P=3$. So $\theta \in {\mathbb{R}}^P$ is a length-$P$ parameter vector. $P$ is the number of weights (and, in a linear model, it matches the feature length).

Each client can train its own copy:

• Client 1 trains on $D_1$ and ends with parameters ${\theta }_1$.
• Client 2 trains on $D_2$ and ends with ${\theta }_2$.
• Client $k$ trains on $D_k$ and ends with ${\theta }_k$.

The server then aggregates those copies into one $\theta$. Aggregation here means a weighted mix of the parameter vectors, not a mix of the datasets. After aggregation, if we could test $\theta$ on every point from every client, the total loss should be small. That is the optimality target: find $\theta$ that minimizes total loss across the clients.

Visual intuition: on a number line of “how wrong the model is on all folders,” we want $\theta$ near the bottom. Each client only sees its own dip. The server never sees the full curve. It only mixes the ${\theta }_k$ that each client found on its own dip.

10.2.4 Student Questions: Is the Goal Network Lag and Quantization?

A natural systems reading is: the hard part is shipping big weight vectors, so maybe the thing we optimize is network lag, and that is why we compress, or quantize, the weights. Those systems issues are real later. They are not the objective of this unit.

A second restatement from the room landed on the right target: we minimize loss with respect to the complete dataset, not only on individual clients. The trigger “transferring weights and network lagging with quantization” is a plausible systems story. It is still the wrong exam answer for “what FedAvg optimizes.”

In production, phones still care about bandwidth, and hospitals still care about round time. Those constraints shape how often you communicate. They do not replace the loss on $D$.

10.3 Global Loss as a Weighted Average of Local Losses

Because data is split, the global loss cannot be computed as one sum over a central table. The classroom replacement is a weighted average of local losses. The weight of client $k$ is its share of the total sample count, $N_k/N$. A client with more points pulls the global loss more. A client with one lonely point does not get a full vote.

Take the same global parameters $\theta$. Send that $\theta$ to client $k$. Client $k$ evaluates loss on $D_k$ only. That number is the local loss of client $k$. Repeat for every client. Mix those local losses with weights $N_k/N$. The mix is the global loss.

10.3.1 Why a Weighted Average

Since data is distributed across multiple clients, the global loss function is computed as a weighted average of each client's local loss. Same $\theta$, which is the global model parameters. Client $k$ uses $D_k$. Average using $N$, the total number of samples, and $N_k$, the number of samples from that client.

If we skipped the weights and treated every client as equal, a tiny client with a wild loss would shove the average as hard as a huge hospital. The $N_k/N$ factor is the fix.

Think of a class average. One quiz taken by 5 students should not count as much as a quiz taken by 45 students. The weight is headcount, not “one vote per room.” The analogy breaks if the 5-student quiz is a different subject: then you are mixing unlike tasks, which is the non-IID issue, not the size issue.

10.3.2 Mathematical Formulation of the Global Objective

Let $F(\theta )$ be the global loss — the scalar we want to drive down. Let $F_k(\theta )$ be client $k$'s local loss on $D_k$. The spoken construction is a weighted average. You take only the sample fraction, the number of samples, $N_k$ divided by $N$:

$$F(\theta )=\sum _{k=1}^K\frac{N_k}{N}{F}_k(\theta )$$

Here $\theta \in {\mathbb{R}}^P$ is the shared parameter vector, $F(\theta )\in \mathbb{R}$ is the global loss, $F_k(\theta )\in \mathbb{R}$ is the local loss of client $k$, $N_k\in \mathbb{N}$ is the local count, and $N={\sum }_{k=1}^K{N}_k$ is the total count. Each weight $N_k/N$ lies in $[0,1]$ and the weights sum to $1$:

$$\sum _{k=1}^K\frac{N_k}{N}=\frac{1}{N}\sum _{k=1}^K{N}_k=\frac{N}{N}=1.$$

The optimization problem is then

$$\underset{\theta \in {\mathbb{R}}^P}{\min }F(\theta ).$$

Limiting case. If $K=1$, then $N_1=N$ and $F(\theta )=F_1(\theta )$: ordinary centralized training. If every $N_k$ is equal to $N/K$, then $F(\theta )=\frac{1}{K}{\sum }_k{F}_k(\theta )$: equal client votes, which is fair only when sizes match.

Domain check. Weights are in $[0,1]$ and sum to 1, so $F(\theta )$ is a convex combination of the local losses. If each $F_k(\theta )\ge 0$, then $F(\theta )\ge 0$.

10.3.3 Local Loss on One Client

How is $F_k(\theta )$ itself built? Walk through every sample inside $D_k$. For each sample $d$, evaluate a pointwise loss $f(\theta ;d)$ with the same $\theta$. Add those pointwise values. The local objective is that average (or that sum; the $N_k/N$ weight already carries the size).

Take all the samples. Within the client dataset you will go through all the samples by name $d$ and the same $\theta$. You will try to compute the loss. Then you will add it.

A standard write-up of that average is

$$F_k(\theta )=\frac{1}{N_k}\sum _{d\in D_k}f(\theta ;d)$$

where $f(\theta ;d)\in \mathbb{R}$ is the loss on one example $d$. The classroom emphasis was summing pointwise losses with shared $\theta$. The $1/N_k$ average is the usual match to a later $N_k/N$ mix.

If $F_k$ is already an average, then the global mix equals the average of $f$ over all points:

$$\begin{array}{rl}F(\theta )& =\sum _{k=1}^K\frac{N_k}{N}\cdot \frac{1}{N_k}\sum _{d\in D_k}f(\theta ;d)\\ & =\sum _{k=1}^K\frac{1}{N}\sum _{d\in D_k}f(\theta ;d)\\ & =\frac{1}{N}\sum _{k=1}^K\sum _{d\in D_k}f(\theta ;d).\end{array}$$

That last line is the mean of $f$ over every local pair. If $F_k$ were a raw sum instead of an average, the same $N_k/N$ mix would be the wrong scale: large clients would be double-counted. The teaching intent is: global loss equals size-weighted mix of local losses, and each local loss uses only local pairs.

Each client computes its own loss using only its local dataset. Then that local loss is reduced by local training. All clients do this. The server never sees $D_k$.

10.3.4 Square Loss and Cross-Entropy

The pointwise loss can be whatever the task needs. Two forms were named.

For a regression-style picture, the square loss was: the label minus theta into $x$, quantity square. With $y_d$ the label of example $d$ and $x_d$ the feature vector of example $d$,

$$f(\theta ;d)={\left(y_d-{\theta }^{\top }{x}_d\right)}^2.$$

Here $y_d$ is a scalar label, $x_d\in {\mathbb{R}}^P$ is the feature vector, and ${\theta }^{\top }{x}_d$ is a scalar score. The phrase “theta into $x$” is the inner product ${\theta }^{\top }{x}_d$. If you store a bias by appending a $1$ onto $x_d$, the same product still holds. In words: $y_d$ is the label, $x_d$ is the data point, every client has pairs $(x_d,y_d)$, and you multiply $x_d$ with the same $\theta$.

Spot-check. Let $\theta =(1,0)$, $x_d=(2,3)$, $y_d=2$. Then ${\theta }^{\top }{x}_d=2$ and $f=(2-2{)}^2=0$. If $y_d=4$, then $f=(4-2{)}^2=4$. Shapes: $\theta \in {\mathbb{R}}^P$, $x_d\in {\mathbb{R}}^P$, product is a scalar, square is a scalar.

For classification, cross-entropy loss is also allowed. We can use any loss. This is not a problem. We are using some loss function. The FedAvg and SCAFFOLD merge rules do not lock you to squares. They lock you to “each client evaluates some $f$ on local pairs, then we mix parameters.”

A common classification form (not required to derive here) is $f(\theta ;d)=-\log p_{\theta }(y_d\mid x_d)$. One-hot cross-entropy $-{\sum }_c{y}_{d,c}\log p_{\theta }(c\mid x_d)$ matches that when $y_d$ is a one-hot label. The merge rule does not care which of those you pick.

The problem type can be regression or classification. The shared object is still one $\theta$. You loop $\theta$ through each client's points. You need that common $\theta$ to do well on all of them.

10.3.5 Worked Fraction Example: Five Local Samples in Fifty

10.3.6 Student Questions and Answers

The next doubt is different: if we add loss across data points, how does that even help with training steps?

Adding loss across data points to find the gradients is a reasonable mix-up: in ordinary centralized training, the sum (or mean) of $f$ is exactly the thing you differentiate. Here the server cannot form that sum. Local FedAvg updates supply the steps. The mixed $\theta$ should make total loss small. That is the re-explanation: summed loss is a score for $\theta$; local updates supply the training direction.

Visual intuition: a bar chart of client sizes. The height of client $k$’s bar is $N_k$. The global loss is a blend of local losses with those heights as weights. A one-sample spike is a short bar. The takeaway: headcount, not hospital count, sets the vote.

10.3.7 Exam Notes

10.4 The FedAvg Algorithm: Local Steps and Weighted Aggregation

FedAvg is the first minimal algorithm for the split-data goal. Multiple clients train. Each has its own parameters ${\theta }_1,{\theta }_2,{\theta }_3,\dots ,{\theta }_K$. The server combines them. The combination taught here is a sample-size-weighted average of the parameter vectors.

A client-sampling rate exists in the full method. It was set aside as “nothing different; leave it,” except for the practical picture: each round, pick a subset of clients $S_t$, such as ten clients, rather than waking every device.

This section is a procedure, not a new definition of $F(\theta )$. The purpose is to train one shared $\theta$ when each $D_k$ stays put. Inputs include the current $\theta$, a batch size $B$, a local step count $L$, and a learning rate $\eta$. Outputs are a new global $\theta$ after each round.

10.4.1 Round Structure and Client Sampling

Inputs named here. Model parameters $\theta$. Batch size $B$. Number of local steps $L$. Learning rate $\eta$ (heard as “eta,” “meta,” “meter,” and “neta”). Loss on a single sample $f(\theta ;d)$. The gradient of that loss.

Server loop, in order:

1. Initialize the global model $\theta$.
2. For each round $t=0,1,2,\dots$:
3. Choose a set of clients $S_t$. Example: ten clients.
4. For each client $k\in S_t$, send the current parameters $\theta$ (and the client id). Also send batch size $B$, local step count $L$, and learning rate $\eta$.
5. Each chosen client runs a client update on its own $D_k$ and returns a new local ${\theta }_k$.
6. The server aggregates the received ${\theta }_k$ into a new global $\theta$.
7. Repeat.

If there are three clients in $S_t$, the same initial $\theta$ goes to client 1, then client 2, then client 3. They train in parallel on private data. They send models back, not mini-batches and not raw rows.

10.4.2 Client Update: Mini-Batches, Gradients, and Learning Rate

The fundamental principle: train the local model with local data. Client $k$ owns $D_k$. A mini-batch $B$ is a small handful of examples drawn from $D_k$, not a packet from the server.

Client update, as a numbered local procedure:

1. Receive the current global $\theta$, plus $B$, $L$, and $\eta$.
2. For local step $\ell =1$ to $L$:
3. Sample a mini-batch of examples from $D_k$. Sample your mini-batch $B$ from the client's local dataset $D_k$.
4. For each sample $d$ in that mini-batch, evaluate the loss $f(\theta ;d)$ and its gradient with respect to $\theta$.
5. Add those gradients.
6. Update $\theta$ by stepping against that gradient, scaled by the learning rate.

The spoken update mixed two ideas at once. Idea A: the number of mini-steps is the total local count divided by batch size, so $L\approx N_k/B$ if you make one pass. Idea B: a written-looking step “theta minus $N_k$ divided by batch size times eta times the (summed) gradient.”

The per-mini-batch step that matches ordinary local SGD is

$$\theta \leftarrow \theta -\eta \cdot \frac{1}{|B|}\sum _{d\in B}{\nabla }_{\theta }f(\theta ;d)$$

where $|B|$ is the batch size (the same capital $B$ in “batch size $B$”), $\eta >0$ is the learning rate, and ${\nabla }_{\theta }f(\theta ;d)\in {\mathbb{R}}^P$ is the gradient of the pointwise loss. Repeat this about $N_k/B$ times to walk the local set.

The literal extra factor that was spoken, “theta minus number of samples divided by capital $B$ into eta into the sum of gradients,” would look like

$$\theta \leftarrow \theta -\eta \cdot \frac{N_k}{B}\sum {\nabla }_{\theta }f.$$

That extra $N_k/B$ is the count of local mini-steps, not a multiplier that should sit inside one SGD step. Read it as “do $N_k/B$ gradient steps,” not as “multiply one step by $N_k/B$.”

Spot-check. Let $P=1$, $\theta =1$, $\eta =0.1$, batch gradients $\{2,4\}$. Mean gradient $3$. New $\theta =1-0.1\cdot 3=0.7$. If you wrongly multiplied by $N_k/B=10$, you would subtract $3$, landing at $-2$, which is a different algorithm.

After the local steps, the client sends its updated ${\theta }_k$ to the server.

10.4.3 Server Aggregation with Sample-Size Weights

The server receives ${\theta }_1,{\theta }_2,{\theta }_3,\dots$ from the chosen clients. It does not treat them as equal votes. It weights each local model by how many samples that client used.

$$\theta \leftarrow \sum _{k\in S_t}\frac{N_k}{N}{\theta }_k$$

Whenever I receive ${\theta }_k$, I multiply by how many samples that client has. Divide by the total number of samples $N$. That is the average. Client 1 used $N_1$ samples. Client 2 used $N_2$. Add those contributions divided by $N$.

This mix is the vanilla FedAvg merge. The global model is formed by taking a weighted average. Aggregating is nothing but taking the thetas and normalizing. Standard form uses the same weighted mean $M_{\mathrm{agg}}={\sum }_i(n_i/N)M_i$ with $N={\sum }_i{n}_i$.

If the sampled set $S_t$ is not the full population, a strict implementation renormalizes by the sum of $N_k$ over $S_t$ only:

$$\theta \leftarrow \sum _{k\in S_t}\frac{N_k}{{\sum }_{j\in S_t}{N}_j}{\theta }_k.$$

The classroom formula used $N$ as the global total. Keep both: the exam story in this unit is “weight by $N_k/N$.” Production code usually renormalizes over the clients who actually answered this round, so the weights still sum to 1.

Boundary check. If every chosen client has the same $N_k$ and $S_t$ is the full set, both formulas reduce to the ordinary mean $\frac{1}{K}{\sum }_k{\theta }_k$.

10.4.4 Worked Example: Local Values 0.8 and 0.5 Mix to 0.59

10.4.5 Why Equal Weighting Hurts

Without the $N_k/N$ weights, every client gets equal importance. Suppose five clients, and one of them trained on only one sample. That one-sample model can have a very high loss. If you add it in as a full vote, it shoves the global $\theta$. If instead you scale by sample count, a spike on one point is diluted across the thousand-point pile.

If a model is trained with only one sample, the loss may be very high. Without averaging, if you add it, that will impact the total loss. If all the samples are, say, a thousand samples, and one sample loss is high, because we are averaging over the number of samples, the impact will be reduced. That is why you use $N_k/N$ times ${\theta }_k$.

The same warning was given twice with the same one-sample-out-of-five picture. Keep both angles: (1) equal votes over-credit tiny clients; (2) size-weighted mix shrinks a one-point spike.

Tiny numeric echo: five clients, four of them send ${\theta }_k=0$ from $N_k=249$ each, one sends ${\theta }_5=10$ from $N_5=4$ (total $N=1000$). Equal mean $10/5=2$. Weighted mix $\frac{4}{1000}\cdot 10=0.04$. The spike is almost gone.

The hoped-for end state: the mixed $\theta$, substituted into the entire dataset, yields a small loss. Each local ${\theta }_k$ should already be a low-loss fit on its own $D_k$. The mix should stay good on the union.

Visual intuition: plot client index on the horizontal axis and ${\theta }_k$ on the vertical axis, with bubble size $N_k$. The FedAvg mix is the size-weighted center of those bubbles. Equal weighting is the center of the bubbles if you ignore size. The takeaway: a tiny bubble should not yank the center.

10.4.6 Student Questions and Answers

The next confusion is about where the mini-batch lives.

The trigger “mini-batch sent by the server to the clients” is plausible if you picture the server as the data owner. In this design the server owns $\theta$, not the rows. Mini-batches are local draws from $D_k$, not server packets.

10.4.7 Industry Applications and Exam Notes

FedAvg is described here as the most popular way to train one shared model across many devices such as phones, hospitals, and banks without collecting private data in one place. Each client trains locally for several steps. Only the updated model is sent. The server averages. Communication cost stays low because data never moves. That is the Gboard-style loop: on-device training, then a weighted mean of weights.

Time cost: each round waits on the slowest chosen client. Space cost: each client stores $\theta$ and $D_k$. The method gets slow when $P$ is huge and every device uploads a full ${\theta }_k$ every round, or when a few devices never finish local steps.

When to use it: start here when clients are roughly similar and you need a simple merge. Alternatives appear when data are unlike (SCAFFOLD) or when you must hide even the updates (differential privacy, later).

10.5 Non-IID Data and the Limits of FedAvg

FedAvg is a first attempt. It can work when clients look alike. It strains when they do not. The name used in the room for “clients look different” was non-IAD data, meaning non-IID data: not independent and identically distributed. IID would mean each client's examples look like draws from the same process. Non-IID means hospital A’s rows do not look like hospital B’s rows.

In an ideal federated setting, local datasets are IID, devices have similar compute, and nobody sabotages the merge. Real hospitals and phones often miss the first of those three. This section is about that miss.

10.5.1 Heterogeneous Clients and Client Drift

In the real world, every client has different-looking data. That is heterogeneous data, also called non-IID data. A hospital in Bangalore does not see the same patients as a hospital in Delhi. Cardiac feature ranges and characteristics can differ by site.

When a client trains locally for many steps, its model slowly gets pulled toward the best answer for its own local data. Client A’s model hugs client A’s records. Client B’s model hugs client B’s records. That pull away from the shared starting point is client drift.

Start from one global model. After local training, client A’s parameters sit over here. Client B’s parameters sit over there. The server average lands in the middle, which may be a poor compromise. Each client drifts toward its own local best. Even though both started from the same $\theta$, local data gives each copy the properties of that client. The average of drifted copies is unstable and slow to settle.

Reference MNIST sketches make the same point with labels: one client holds digits 0–4, the other holds 5–9. Local tasks are no longer the same 10-class problem. Repeated averaging then pulls each model toward the wrong star.

10.5.2 Hospital Bias Walkthrough

Imagine federated training across hospitals for a condition that is not purely an eye disease. One round of sites is dominated by patients who mostly have eye problems and few other conditions. The local datasets are then biased. Aggregating those local models can bake in the bias.

That is non-IID structure, not a coding bug. One site’s cardiac features live in one numeric range. Another site’s live in another. We should not expect a 100% accurate global model. Random guessing in a balanced two-class sketch sits near 50%. The working aim named here is in the 70% to 80% band, not perfection.

We may not expect it to be a hundred percent accurate. We are aiming at some 80%, not a random 50%. We aim till 70%, 70%, and so on. FedAvg is not very accurate in these terms, but it can still lead to something usable. It is the first attempt. Later work improves it. SCAFFOLD is one of those improvements. Many other aggregation proposals exist, including a later family named FedBSS, which was only previewed.

Visual intuition: two arrows in the plane of model parameters. One arrow points at “Bangalore cardiac range.” The other points at “Delhi cardiac range.” FedAvg draws the midpoint. If the arrows are almost opposite, the midpoint is not a hospital that exists. The takeaway: unlike data yields unlike local optima; the mean need not be a good model.

10.5.3 What FedAvg Cannot Guarantee

Three failure pictures now sit on the table:

1. Size imbalance. A one-point client can wreck an unweighted mix. FedAvg’s $N_k/N$ weights reduce that.
2. Progress imbalance. Some clients have already minimized; others have not. FedAvg still averages all of them and still asks the healthy clients to compute again.
3. Distribution imbalance. Non-IID drift. FedAvg averages drifted models and can fail to converge, especially if local optima point in very different directions, even roughly orthogonal directions.

SCAFFOLD attacks picture 3 by measuring drift and subtracting it.

On a sketch where two local gradients are orthogonal, the average step can cancel useful progress on both axes. That is why “just average whatever arrived” is not enough when clients are unlike.

$$\frac{N_k}{N}$$

still appears in the merge. It is the size fix from Section 10.4. It is not a drift fix. A huge Delhi hospital still pulls $\theta$ toward Delhi patients.

10.5.4 Student Questions and Answers

The student reading is right: aggregating biased eye-problem hospital datasets can be a bad idea. It is a data-distribution problem, not only a merge-formula problem. Treat it as limited-accuracy FedAvg, not as a coin-flip success and not as a coding bug.

10.5.5 Industry Applications

A hospital in Bangalore versus a hospital in Delhi is the running geographic picture for heterogeneous patients. Cross-hospital training without shipping records is the reason to federate at all. The cost of that choice is drift. Phones and banks have the same pattern: each device or branch is its own distribution. Keyboard logs differ by language and slang. Branch ledgers differ by city.

10.6 SCAFFOLD: Measure Drift, Then Subtract It

SCAFFOLD’s teaching slogan is short: measure the drift, then subtract it. Along with the model, the server and each client keep a small extra vector called a control variate. Think of that extra vector as a note. The note says: your local data tends to pull you in this direction, while the whole group actually needs to move in that direction.

FedAvg is simple and cheap to communicate. SCAFFOLD spends a bit more state — one control vector on the server and one on each client — to cancel the systematic local pull.

This is a procedure that sits on top of the same federated contract. Data still never moves. What changes is the local step and the extra note that travels with the model.

10.6.1 Client Drift After Local Training

Picture the global model as a point. Client 1 trains and walks toward its local best. Client 2 trains and walks toward a different local best. The average of those two walked points is not the true joint step. If you skip the scaffold-style correction, the path zigzags. The server model sits in one place. Client models sit elsewhere. Averaging them writes a new server model that still is not the joint optimum.

The job is to reduce that drift. When local training would shove $\theta$ too far, we subtract a correction so $\theta$ is not allowed to run beyond a band. You are not allowing the theta values to move beyond a certain value. If you allow it, it may deviate too much. If you go beyond, subtract the value. You should not deviate much. The drift should not happen.

A demo contrast was shown. FedAvg’s clients send raw updates; convergence takes time and may never arrive. SCAFFOLD’s clients send values after a drift correction; the path settles because you are subtracting the drift, so there is not much deviation.

10.6.2 Control Variates as Direction Notes

Let $x$ be the global model on the server (the same object we called $\theta$ before; SCAFFOLD’s algebra used $x$ and $y_i$). Let $c$ be the global control variate — the group direction. Let $c_i$ be client $i$’s control variate — that client’s stored local direction or bias. Let $y_i$ be the local working copy of the model on client $i$.

Shapes match the model. If $x\in {\mathbb{R}}^P$, then $c\in {\mathbb{R}}^P$, $c_i\in {\mathbb{R}}^P$, and $y_i\in {\mathbb{R}}^P$. A model with about one lakh (100,000) parameters has a control vector of the same length. Weight number 5377 of the model has its own personal correction entry: component 5377 of $c_i$. Each parameter has two parts to watch — how this parameter is deviating for this client, and how it is deviating from the global point of view.

Notation: texts keep $x$ for the server model and $\theta$ for FedAvg. This section follows the SCAFFOLD letters. They name the same kind of object: a length-$P$ weight vector.

What is stored in $c_i$? The gradient of client $i$’s local loss, averaged over client $i$’s whole dataset, at the current global model. So $c_i$ tracks, per parameter, how much local data wants to adjust that weight.

$$c_i\approx \nabla F_i(x)=\frac{1}{N_i}\sum _{d\in D_i}\nabla f(x;d)$$

Here $F_i$ is client $i$’s local loss, $D_i$ is its full local set, $N_i$ is its local count, and $x$ is the current global model. Take the current global model $x$. Go through the samples from the dataset. Compute the local loss. Take the average. Store that in $c_i$.

The global $c$ is the matching group object: an average of those local directions, so it points where the whole set of clients is moving.

$$c\approx \frac{1}{K_{\mathrm{clients}}}\sum _{i=1}^{K_{\mathrm{clients}}}{c}_i$$

when every client participates and each $c_i$ is already a mean gradient. In the running algorithm, $c$ is maintained by mixing uploaded control changes, not by a fresh full-data pass on the server.

Domain check. $\nabla F_i(x)$ has the same shape as $x$. A probability loss can make some coordinates of $c_i$ large; nothing forces $\Vert c_i\Vert =1$. It is a direction and a magnitude, not a forced length-one vector.

10.6.3 The Corrected Local Update

Each round, the server samples some clients and sends them both $x$ and $c$. On a client:

1. Initialize the local model: $y_i\leftarrow x$. Whatever it receives, it copies into $y$.
2. Repeat local steps $K$ times (the local loop count; this $K$ is local steps, not the number of clients used earlier for FedAvg).
3. Each local step: draw a mini-batch, compute a mini-batch gradient $g$ of the local loss at $y_i$, then update with a correction.

The heart of SCAFFOLD is this line. Read the bracket as: raw local direction, minus the client bias, plus the group direction.

$$y_i\leftarrow y_i-\eta \left(g(y_i)-c_i+c\right)$$

Here $\eta >0$ is the local step size (the “constant”), $g(y_i)$ is the mini-batch gradient at the current local model, $c_i$ is the client control, and $c$ is the global control.

$y_i$ equals $y_i$ minus a constant times (gradient of that model minus $c_i$ plus $c$). Raw local direction is how my gradients are moving. I subtract the bias. I add the overall group direction — how all the other clients are moving. I subtract my own drift so I stay in line with the global moving trend. Model parameters are updated by considering the overall global trend and subtracting the current model bias.

Why this cancels drift at the source: $c_i$ cancels the systematic pull of my data. Adding $c-c_i$ removes what my data always says and inserts what the group says. If the local gradient $g$ is the usual SGD direction, then

$$g(y_i)-c_i+c=g(y_i)+(c-c_i).$$

The term $c-c_i$ is the correction term. The local dashed gradient may point at that client’s private minimizer $x_1^{\star }$. The correction shoves the step back toward the global (blue) trend.

10.6.4 Refreshing Client Controls and Sending Differences

After the local loop, the client refreshes $c_i$. Two options were named.

Option A. Set the new client control from a gradient of the global model, spoken as “$g$ of $x$”:

$$c_i^{+}\leftarrow g(x).$$

That matches the paper’s Option I: evaluate the local mean gradient at the received server model $x$. It needs an extra pass over $D_i$ (or a large batch) at $x$.

Option B. A control-state update spoken as: $c_i$ minus global plus one over $K$ eta times $x$ minus $y_i$:

$$c_i^{+}\leftarrow c_i-c+\frac{1}{K\eta }(x-y_i).$$

Here $K$ is the local-step count, not the FedAvg client count. This is the paper’s Option II. It reuses the displacement $x-y_i$ already caused by the $K$ corrected steps, so you do not form a fresh full-batch gradient.

Why Option B is algebra, not magic. If every local step used the same corrected direction $g-c_i+c$ and we pretend $g$ is constant for a sketch, then

$$y_i\approx x-K\eta \left(g-c_i+c\right),$$

so

$$\frac{1}{K\eta }(x-y_i)\approx g-c_i+c,$$

and

$$c_i-c+\frac{1}{K\eta }(x-y_i)\approx g,$$

which is a stand-in for a new local control. Real mini-batch $g$ changes each step; Option B is still the cheap estimator used in practice.

Then the client does not upload the full $y_i$ and full $c_i$ as absolute portraits. It uploads differences:

• Parameter change: $y_i-x$
• Control change: $c_i^{+}-c_i$

The client uploads differences, not absolute values; $c_i^{+}$ minus $c_i$, and the difference in the parameters. Those values are used to compute the new $x$ and $c$. Every client sends the corrections only. Once the server receives the corrections, it updates $c$ and $x$. $x$ is the model. $c$ is the global correction state.

The server also has a global step size ${\eta }_g$ among the named inputs. Let $S$ be the clients who answered this round, and let $N_{\mathrm{clients}}$ be the total client population. A standard mixing rule is

$$\begin{array}{rl}x& \leftarrow x+{\eta }_g\cdot \frac{1}{|S|}\sum _{i\in S}(y_i-x),\\ c& \leftarrow c+\frac{|S|}{N_{\mathrm{clients}}}\cdot \frac{1}{|S|}\sum _{i\in S}(c_i^{+}-c_i).\end{array}$$

The classroom did not dictate those two lines in full. The teaching content is that the uploaded $\Delta y=y_i-x$ and $\Delta c=c_i^{+}-c_i$ are the only ingredients for the server’s update of $x$ and $c$. Keep the professor’s story (send differences; server refreshes $x$ and $c$) and treat the display above as the usual completion of the missing algebra.

10.6.5 Per-Parameter Corrections in a Large Model

Do not picture one scalar “drift number” for the whole network. Picture a parallel correction for every weight.

Suppose the current combined view of one weight’s trend across clients looks like $+5,-5,-6,-3.2$ on different coordinates of the story. Client 1’s own $\theta$ trend on a coordinate might be different from the average of all clients. SCAFFOLD keeps, inside $c_i$, how much to correct each coordinate.

If weight 5377 on the client wants to move by $-5$ relative to the group, that $-5$ lives in component 5377 of $c_i$. The ordinary gradient step on that weight would be the usual

$${\theta }_1\leftarrow {\theta }_1-\eta \frac{\partial L}{\partial {\theta }_1}.$$

SCAFFOLD still uses a gradient, then adds the stored correction so that the step follows the group, not the private yank.

Complexity. Extra memory is one vector of length $P$ on the server and one on each client. Extra communication is one more length-$P$ difference $\Delta c$ beside $\Delta y$. Local time grows with $K$ mini-steps, same as FedAvg, plus a control refresh.

10.6.6 Student Questions: What Is the Global Direction?

The trigger used a unit vector picture. That is a fair geometric instinct (keep only direction). The stored $c_i$ is not forced to length one. It is the (mean) local gradient, so it keeps magnitude. The global direction is still “average of those controls,” not a separate oracle.

A second pass in the same exchange: five clients including you, your deviation $-5$, others $2,3,4,5$. The spoken average was “seven nine by five,” about $9/5=1.8$, rounded in speech to “one point something” and then “1.5.” Your trend is $-5$. You subtract your effect. You add the leftover global trend, not the drift that happened in your case.

10.6.7 Exam Notes

10.7 Numerical Walkthrough of the SCAFFOLD Correction

The correction is easier with three numbers than with a full network. The room worked the same idea more than once with slightly shifting figures. Both passes are kept. They teach the same rule: do not take the old value minus the raw local gradient; take a step that uses global trend minus local trend so the new value stays nearer the group.

These figures are teaching sketches. They are not a polished least-squares fit of a real network, and they are not a promise that one exam paper will ask you to grind many decimals.

10.7.1 Setup With Three Parameters

Let the current global model have three coordinates

$$x=(5,6,7).$$

So ${\theta }_1=5$, ${\theta }_2=6$, ${\theta }_3=7$. These are the current global parameters.

Client 1’s local trend on those three coordinates was first spoken as $(-2,+1,-3)$. Other clients were moving positive, with a second client spoken as $(+3,+5,+6)$. A third client was added as also moving positive. The average of the first coordinate’s story was spoken as “7 minus 2 is 5; 5 by 3, say 1.5.” That 1.5 is the global trend $c$ on the first coordinate in this sketch. A strict mean of $5/3$ is about $1.67$; speech rounded to $1.5$. The mix of client gradients was informal.

A later pass used client values $(-2,+3,+4)$ on the three coordinates, and again a global first-coordinate average of $1.5$ from “C1, C2, C3.”

Keep $1.5$ as the classroom global trend on the first weight, and keep $-2$ as this client’s local trend on that weight.

Matching informal group trends for the other coordinates in the first pass: “6 plus 6 is 12, 12 by 3 is 4” and “4 plus 6 is 10 by 3, something like 3.3.” Those were not fully turned into side-by-side landings.

10.7.2 Naive Local Step Versus Corrected Step (First Pass)

That pass treated the local trend as a magnitude $2$ in the subtraction $1.5-2$. It is not a unique algebra line. Subtracting local from global for each $\theta$ coordinate still matches the slogan, because each of $c_1,c_2,c_3,\dots$ holds that client’s gradient on that coordinate.

10.7.3 Second Classroom Pass With the Same Idea

The second pass used the same global first weight $5$ and the same local trend $-2$, and it named the naive landing $7$ again. This pass is a more literal “step size equals $c-c_i$” picture.

The two landings disagree because the spoken algebra shifted: first pass $5.5$, second pass $1.5$. Keep both as they were taught. Do not pick a silent winner.

A third, standard SCAFFOLD line on the same numbers, for reconciliation only. If the mini-batch gradient equals the stored local control, $g=c_i=-2$, and $c=1.5$, $\eta =1$, then

$$\begin{array}{rl}g-c_i+c& =-2-(-2)+1.5=1.5,\\ y& \leftarrow 5-1\cdot 1.5=\mathrm{3.5.}\end{array}$$

That $3.5$ is another landing. It is the ordinary paper update when local gradient matches $c_i$. The exam story in this unit is still the comparison against naive $7$, not a demand that you reproduce $3.5$ unless the question writes $g-c_i+c$ explicitly.

10.7.4 Five-Client Net-Effect Example

Another net-effect sketch used five clients and one parameter.

• Your deviation: $-5$
• Other spoken values: $2,3,4,5$
• Sum of the five numbers $-5+2+3+4+5=9$
• Average $9/5=1.8$, spoken as “seven nine by five,” “one point something,” then “1.5”

The group trend is about $1.5$ to $1.8$. Yours is $-5$. You do not add $-5$ to $\theta$. You subtract your effect and add the leftover group trend. You are adding the global trend to your parameter, not the drift that happened in your case.

If all five were treated as controls $c_j$, the global $c$ is their average. Your update uses $c-c_i$, which removes the $-5$ pull and leaves the net group direction:

$$c-c_i\approx 1.8-(-5)=\mathrm{6.8.}$$

You do not need that $6.8$ as a memorized exam number. You need the direction of the story: drop your $-5$ yank; keep the group mean near $1.5$–$1.8$.

10.7.5 What the Demo Curves Show

A side-by-side path was described. At first both FedAvg and SCAFFOLD deviate. Then SCAFFOLD converges. FedAvg keeps wandering and is not able to converge in the demo.

Why does the FedAvg curve keep moving? Each dataset points in a different direction. The directions may even be orthogonal. The datasets themselves stay the same in both methods. The only difference is the parameter update.

• FedAvg: no restriction; whatever the client sends, average it.
• SCAFFOLD: before the client sends, it is given the global trend; it nullifies local trend against global trend; it adds only the remaining group piece.

Because almost all clients remove their own drift, the path can settle. The local dashed gradient still points at the private star. The correction term $c-c_i$ brings the step back toward the global trend, again and again.

Visual intuition: two traces versus round index (horizontal axis) and a one-number stand-in for “how far the server model still moves” (vertical axis). FedAvg’s trace keeps oscillating. SCAFFOLD’s trace rises at first, then flattens. Landmark: the flattening is the correction doing its job, not a change of dataset.

10.7.6 Student Questions and Answers

10.8 When FedAvg and SCAFFOLD Match, and When a Global Model Is Not the Goal

Two closing questions matter in practice. First: when is SCAFFOLD the same as FedAvg? Second: what if you never wanted a shared global model at all, because your phone should follow your vocabulary?

The short answer is no: the extra note is for unlike pulls. The rest of this section makes that algebraic, then separates a shared $\theta$ from a personal on-device product.

10.8.1 Homogeneous Data

FedAvg and SCAFFOLD line up when the dataset is homogeneous — clients look like each other, IID rather than non-IID. If every local distribution matches, local drift is not a different direction from the group. The extra correction has nothing special to cancel.

When the dataset is homogeneous, FedAvg and SCAFFOLD will be the same.

That is the statistical test. Alike clients, alike local gradients, alike controls. Then measuring drift and subtracting it does not change the step.

10.8.2 Zero Correction When Local and Global Controls Match

A sharper algebraic condition was accepted in the room: the two methods match when client drift matches server drift, that is, when $c_i$ and $c$ are the same.

Look at the extra piece that SCAFFOLD adds, $c-c_i$. If that resultant is zero, the same parameters are used as in an uncorrected step. If all clients move in the same direction, there is no global-versus-local split. Then

$$c-c_1=0,c-c_2=0,c-c_3=0$$

and the SCAFFOLD update collapses to ordinary local gradient descent plus ordinary averaging. If every three, or all, of the parameters travel toward the same direction, then there is no global drift and all the locals are the same.

Write the collapse in one line. Start from

$$y_i\leftarrow y_i-\eta \left(g(y_i)-c_i+c\right).$$

If $c_i=c$, then $-c_i+c=0$, so

$$y_i\leftarrow y_i-\eta g(y_i),$$

which is the uncorrected local SGD step used inside FedAvg. The server mix can then be the same weighted average of the $y_i$.

So two verbal tests:

1. Homogeneous data (clients statistically alike).
2. $c_i=c$ (local control equals global control), so the correction is nullified.

10.8.3 Personal On-Device Models Versus a Shared Global Model

A systems question pushed against the whole global-model use case. On a mobile device, you may want next-word guesses from your vocabulary and your stock of words. You may not want a global model. You may not want to average at all. You want a model that is good on the edge case that is you.

The reply does not deny that personalization exists. It blocks a false split between “global model” and “local model.” There is no global model built by ignoring local models. The global model is built on top of local models. Local client models are client-specific in the sense that they learn the same $\theta$ on that client’s samples. The final $\theta$ still has to work with those local samples.

Look back at the loss. There is no separate global loss computed on secret central rows. Local loss is reduced on local rows. ${\theta }_1$ is best for client 1’s data. ${\theta }_2$ is best for client 2’s data. Those ${\theta }_k$ are sent and aggregated. The first condition is: there is nothing like a global model that performs worst on the client set. The mixed $\theta$ is scored by the same local losses.

You cannot come up with a global model by ignoring the local model. The global model is built on top of the local model. When we say the final model, that final model is $\theta$, and that $\theta$ has to work with the local samples of the local client.

This does not claim that one mixed $\theta$ is always the product you want on a keyboard. It claims that the FedAvg/SCAFFOLD object, if you are training a shared $\theta$, is already tied to local performance through local loss. A fully personal model that never averages is a different product. It was flagged as a good question to reopen with more time.

Reference treatments call that other product personalization: fine-tune the shared model on local logs, accepting a trade-off between “good on me” and “good on everyone.” That is not the same as refusing to average during the shared training loop.

Visual intuition: two stacked boxes. Bottom box: local losses on local rows. Top box: mixed $\theta$. Arrows go up from every local box into the mix, then back down as the next starting $x$. There is no side door labeled “global loss on hidden central data.”

10.8.4 Student Questions and Answers

The second question is the keyboard use case, not a repeat of the algebra.

The trigger “words predicted from my vocabulary on a mobile device” is a real product wish. The resolution is not “you must average forever.” It is: you cannot build that shared global model by ignoring the local models. A shared $\theta$ is built on top of local models and local loss. A never-averaged on-device model is another product.

10.8.5 What Comes Next

Advanced aggregation methods continue after SCAFFOLD. FedBSS was named as one of the next algorithms in the prepared sequence. Differential privacy — how to apply it in this setting — was named as a following topic. The same control-variate picture will be reused rather than replaced.

Mobile next-word prediction is the concrete personalization story. Hospitals and banks remain the concrete shared-model story. Both live under the same constraint: raw records stay on site.

Exam Guidance Summary

There will not be mathematical problem-solving questions of the long numeric kind. The paper is set to test whether you understand the algorithms. How does FedAvg work? How does SCAFFOLD work? What is the role of each piece? When is the correction nullified? What are the uploaded differences, spoken as $dx$ and $dc$ — the change in the model and the change in the control?

Do not go too deep into mathematics. That depth is not required here. If a numeric-style problem were ever used, similar problems would be discussed first. The planned emphasis is conceptual: steps, roles, and the homogeneous / $c_i=c$ collapse.

Open-book materials will be watermarked copies of the course visuals. Several more sessions remain in the sequence (spoken as six more, then corrected to about five more). Use that time on algorithm flow, not on replica algebra.

Key Industry Applications

Phones, hospitals, banks. Federated averaging trains one shared model across many devices without collecting private data in one place. Only updated models move. Communication stays comparatively cheap. The same loop is the industrial story behind on-device keyboards and cross-silo medical or financial models.

Vertical hospital features. Eye, cardiology, and skin views of patients $P_1$ and $P_2$ show how overlapping fields can fill missing fields. Industry uses vertical federated learning to combine views, train, and complete missing features without sending a joined raw table.

Cross-city hospitals. Bangalore versus Delhi patients is the picture for non-IID medical data. Cardiac ranges differ. Expect roughly 70%–80% utility, not 100%, from vanilla FedAvg.

Biased specialty sites. A network of eye-heavy hospitals can poison a naive aggregate. That is a data-distribution problem, not just a merge-formula problem.

On-device keyboards. Next-word prediction from a personal vocabulary is the case where a user may not want a global average at all. The shared-model math still insists that any global $\theta$ be built from local losses; a purely personal non-averaged model is a different product.

Follow-on production tools. SCAFFOLD is one repair for drift. FedBSS was named as a later algorithm in the same industrial line. Differential privacy was named as the next protection to apply on top of these training loops.