Distributed Machine Learning

SCAFFOLD and FedProx: Controlling Client Drift

Published: 2026-09-11

Level: postgraduate

Audience: Postgraduate students in Distributed Machine Learning

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

Hospitals, Non-IID Data and When Vertical Partitioning Appears — covered in Lecture 1
Non-IID Data: Same Features Versus Mixed Features — covered in Lecture 3

A family of aggregation methods takes weights or gradients from many clients and builds one global model — the shared parameter vector that the server keeps. How those updates are mixed decides how well the combined model works.

SCAFFOLD, FedProx, and a related method named FedDSS each target a different failure mode of that mix. This note goes deep on SCAFFOLD and FedProx. Both try to stop a client from wandering too far from the pack.

The path is concrete. First we see why a plain average of local models can fail when each device sees a skewed slice of the world. Then we rebuild gradients as arrows, add SCAFFOLD’s control-variate correction, walk a full two-client round with real numbers, and only after that switch to FedProx’s rubber-band penalty. Keep the same symbols as you go: $x$ for the server model, $y_{i}$ for a client copy, $c_{i}$ and $c$ for the two control arrows, and $μ$ for FedProx’s leash.

11.1 Combining Client Models When Data Is Not Shared

The server does not train on raw samples. It receives weights from client 4, client 3, client 2, and client 1, or it receives gradients. Then it must form one global model — the shared parameter vector that the server keeps. The quality of that mix is the quality of the system.

If each hospital, phone, or camera only ever sees its own slice of the world, can you still average their models and trust the result? The mix can look tidy and still point the shared model in the wrong direction.

11.1.1 Why Simple Mixing Is Not Enough

Picture four clients standing around a shared model. Client C1 may hold only cat images. Client C2 may hold only dog images. Client C3 may hold elephant images. Each local trainer becomes an expert on its own animals. That local skill is useful. It is also dangerous.

Four phones join a photo classifier round. C1 has 200 cat shots and almost no other animals. C2 has 180 dog shots. C3 has 150 elephant shots. C4 has a mixed album. After many local steps, C1’s weights are a cat specialist. C2’s weights are a dog specialist. If the server only averages finished weights, the shared model can tilt toward whichever specialist shouted loudest. The goal is not to freeze every client in place. The goal is that no client should deviate — wander far in parameter space — from the other clients and from the global model. C2, C3, and C4 should sit near one another. C1 should sit near them too.

Think of four hikers on a foggy hill, each holding a private map. A gradient (the steepest-uphill arrow a hiker feels underfoot) tells each hiker which way loss rises fastest on their own map. If C1’s map is “cats only,” that arrow is a cat arrow. Averaging the hikers’ final positions is not the same as walking a path that is fair to dogs and elephants. The analogy breaks when the “hill” is a high-dimensional weight space: you cannot see the arrows with your eyes, but the same pull-away story still holds.

Researchers split this pain into pieces and named an algorithm for each piece. SCAFFOLD handles client drift. FedProx handles another control idea built on distance. FedDSS handles yet another feature. All of them are recipes for combining local models into a global model.

11.1.2 Non-IID Data and Client Drift

Federated learning trains one model while data stays on devices. The server never sees the raw photos or hospital records. Clients are heterogeneous: they are not copies of one another. Each client trains on a different slice of the world.

The data are non-IID — not independent and identically distributed. Independent would mean one sample does not secretly dictate the next. Identically distributed would mean every client draws from the same pattern and the same label mix. You cannot expect that on every phone or hospital. One ward sees one disease mix. Another ward sees another mix.

That mismatch is what produces client drift. Drift here is not a vague vibe. It is a geometric fact: after many local steps, two clients’ weight vectors no longer point the same way. A cat-only client keeps pushing weights in a cat direction. A dog-only client keeps pushing in a dog direction. The longer each client trains alone, the farther those two arrows separate.

SCAFFOLD’s job, in one sentence, is to reduce that drift while still averaging.

Scope: Ordinary averaging works well when local datasets are close to IID, devices have similar compute, and nobody is trying to poison the mix. Those three “ideal federated” conditions are the usual baseline in textbooks. When any one of them fails, a sample-count weighted mix can still run, but the shared model can diverge from what a pooled dataset would have learned. This section is about the non-IID failure, not about attackers.

Imagine a scatter plot whose x-axis is “how cat-like the weights are” and whose y-axis is “how dog-like the weights are.” After one local epoch, the four clients are a tight cloud around the global point. After many local epochs, C1 has marched far along the cat axis and C2 far along the dog axis. The takeaway: extra local work without a leash turns specialists into outliers.

11.1.3 Ordinary Federated Averaging

Federated averaging (FedAvg) is the simple baseline. It does not try to diagnose drift. It takes whatever weights arrived and averages them, scaled by how many samples each client trained on.

Let client $i$ hold $n_{i}$ training samples and local weights $w_{i}$ . Let the total sample count be $N = n_{1} + n_{2} + n_{3}$ when three clients report. The verbal description is: take the number of samples a client trained on, divide by the total number of samples, and multiply by that client’s weights. For client 2 that factor is $n_{2} / N$ . For client 3 it is $n_{3} / N$ .

The mixed model for the three-client picture is

$w \leftarrow \frac{n _{1}}{N} w_{1} + \frac{n _{2}}{N} w_{2} + \frac{n _{3}}{N} w_{3}$

with $N = n_{1} + n_{2} + n_{3}$ . Here $w_{i}$ is the weight vector (or a toy scalar) on client $i$ , and $w$ is the new global model. Bounds here follow the three-client picture used in the explanation. Later work keeps the same weighted-average idea for any number of clients $m$ :

$w \leftarrow i = 1 \sum m \frac{n _{i}}{N} w_{i}, N = i = 1 \sum m n_{i} .$

Each coefficient $n_{i} / N$ is a fraction in $[0, 1]$ and the coefficients sum to $1$ , so this is a convex combination of the local models.

Suppose three clinics report scalar toy weights $w_{1} = 1.0$ , $w_{2} = 3.0$ , $w_{3} = 5.0$ with sample counts $n_{1} = 100$ , $n_{2} = 50$ , $n_{3} = 50$ . Then $N = 200$ , so

$w = \frac{100}{200} (1.0) + \frac{50}{200} (3.0) + \frac{50}{200} (5.0) = 0.5 \times 1.0 + 0.25 \times 3.0 + 0.25 \times 5.0 = 0.5 + 0.75 + 1.25 = 2.5 .$

Sense-check: clinic 1 has half the data, so the mix sits closer to $1.0$ than to $5.0$ . An unweighted mean would have been $3.0$ . Sample-count weighting is doing real work.

FedAvg is easy. It also ignores the hard part: a client that saw a weird slice of data can yank the average off course. That is why a plain average can be unsafe when labels are skewed.

A second numeric picture makes the yank visible. Keep the same $n_{i}$ but let the cat clinic’s scalar weight run to $w_{1} = 20$ while the others stay at $3$ and $5$ . Then $w = 0.5 \times 20 + 0.25 \times 3 + 0.25 \times 5 = 10 + 0.75 + 1.25 = 12$ . The shared model followed the cat specialist even though two clinics never saw cats.

A common trap is to treat FedAvg as “just the mean of models.” If you forget the $n_{i} / N$ weights, a tiny clinic with 10 samples votes as hard as a hospital with 10,000. Another trap is to think a longer local run is always better. Extra local steps can grow drift. A third trap is to mix gradients and weights as if they were the same object: FedAvg as written here averages parameters $w_{i}$ , not raw mini-batch gradients.

11.1.4 Student Questions and Answers

The opening question is the whole course in one line: the server has to combine what it got, and the mix quality is the model quality.

Q: At a high level, what are SCAFFOLD, FedProx, and FedDSS doing when the server gets weights from many clients?

A: The server must combine those weights or gradients into one global model. The mix quality is the model quality. Clients may each be experts on different data, such as cats versus dogs versus elephants. The shared aim is that a client should not deviate too far from the others or from the global model. Each method attacks a different piece of that mixing problem. SCAFFOLD targets drift. FedProx targets a different control idea. FedDSS targets another feature.

11.1.5 Industry Applications and Exam Notes

On-device training for photos is the running picture. One phone may mostly shoot pets of one kind. Another phone may mostly shoot a different kind. A hospital site may see one disease mix. Another site may see another mix. Keyboard-next-word models on phones hit the same skew: one user types sports slang, another types medical terms. Weighted averaging alone does not fix that skew. Production federated stacks still start from FedAvg because it is simple, then swap in a drift-aware mix when the label mix is badly split.

This sits in the broader field as the aggregation layer of federated learning: after local training, before the next broadcast. Everything later in this lecture is a replacement rule for that one mix step.

Exam note: Be ready to write the FedAvg mix $w \leftarrow \sum_{i} (n_{i} / N) w_{i}$ and to say in plain words why non-IID client data makes a plain average unsafe. Name the cat / dog / elephant picture as the reason the specialists can pull the global model off course.

FedAvg is the baseline we now improve. SCAFFOLD will interfere during local training instead of waiting until the weights are finished.

11.2 SCAFFOLD: Stochastic Controlled Averaging

SCAFFOLD stands for stochastic controlled averaging. It is still an averaging method: the server receives information from many clients and averages. The new idea is a control on the average so that local drift is not left free.

“Stochastic” here points at the noisy mini-batch gradients. Each pack of samples gives a slightly different arrow. “Controlled” points at the extra state that remembers the usual arrow and edits the noisy one.

FedAvg waits until local training is done, then blends finished weights. Why not grab the steering wheel during those local steps, while the client is still walking?

11.2.1 What “Controlled” Adds on Top of Averaging

Ordinary FedAvg treats every uploaded weight as a fair vote, scaled only by sample count. SCAFFOLD keeps a memory of how each client usually moves, and a memory of how the whole population usually moves.

Those memories are control variates. A control variate (an extra vector that tracks a typical gradient direction) lives in the same shape as the model weights. There is one vector $c_{i}$ on client $i$ and one vector $c$ on the server. They describe typical gradient direction. The client uses them to correct the current mini-batch gradient before it walks too far.

A useful picture: FedAvg waits until training is done, then blends finished weights. SCAFFOLD interferes during local training. Each local step is not “whatever my batch wants.” It is “what my batch wants, after I remove my usual private tilt and face the group tilt.” That is why the method can reduce drift even when every client still trains only on its own data.

There is no peer-to-peer link among clients. A client cannot see the other clients’ typical directions. So the server must ship the global control vector down with the model. The client ships back changes, not the full new vectors. Sending a difference is lighter than sending a full new model plus a full new control if you already know the old values on the server, and it matches the later server step, which averages those differences. That communication pattern is part of the method, not an implementation footnote.

Later we will see a second method, FedProx, that tries to solve the same drift pain without those extra vectors. Keep that contrast in mind: SCAFFOLD buys directional control by storing and shipping extra state.

Assumption: SCAFFOLD’s correction is honest about what the client can see. The client already stores $c_{i}$ . It does not see other clients. The global $c$ must arrive from the server. If the server sent only the model $x$ , the client could not form the group direction on its own.

Picture two arrows taped to a compass. One arrow is “how I usually walk.” The other is “how the group usually walks.” SCAFFOLD rewrites the current step by swapping the first arrow for the second. The analogy breaks because the arrows are as long as the whole model — millions of coordinates — not a pocket compass.

11.2.2 Symbols Used Throughout SCAFFOLD

In the one-number toy used later, global weights start at $x = 10$ . That single number is a stand-in for a full weight vector. Every add and divide in later sections is the same operation you would run coordinate-wise on a real network.

Let $x$ be the global model weights on the server. In the one-number toy, $x$ starts at $10$ . Let $y_{i}$ be client $i$ ’s local copy of the weights after local training. Let $c_{i}$ be client $i$ ’s control variate: a running picture of that client’s usual gradient direction. Let $c$ be the global control variate: the average direction preferred by the clients as a group. Let $g_{i}$ be the current mini-batch gradient on client $i$ . Let $η$ be the local learning rate — the step size. Let $K$ be the number of local steps in a round. Let $S$ be the set of clients that take part in this round. Let $N$ be the total number of clients in the system, including those that sat out.

The verbal description of the control idea is: $c_{i}$ is client $i$ ’s usual local gradient. Gradients are directions in which loss rises fast. If you watch many mini-batches, that direction settles toward a typical value for that client.

The local corrected step, written once so the symbols sit together, is

$y_{i} \leftarrow y_{i} - η (g_{i} - c_{i} + c) .$

Each of $x$ , $y_{i}$ , $c_{i}$ , $c$ , and $g_{i}$ has the same shape. If the model has three weights, each of these is a length-3 vector.

The standard algorithm uses the same symbols. Texts sometimes write a server step size $η_{g}$ in front of the averaged model change. In the numeric round later, that server step size is $1$ .

11.2.3 Student Questions and Answers

The correction uses three arrows: the current batch, the client habit, and the group habit. Only the server can form the third.

Q: Why must the server send the global control $c$ down to the client, not only the model $x$ ?

A: The correction uses the current batch gradient, minus the client’s own usual direction, plus the global usual direction. The client already stores $c_{i}$ . It cannot see the other clients, so it cannot form $c$ by itself. There is no peer-to-peer channel. The server must send both the model and the global $c$ .

11.2.4 Exam Notes

Do not treat $c_{i}$ as “the current mini-batch gradient.” $g_{i}$ is the current pack. $c_{i}$ is the memory of many packs. Do not treat $N$ and $∣ S ∣$ as the same count: $N$ is everyone in the fleet, $∣ S ∣$ is who joined this round. Do not skip the downlink of $c$ ; a client with only $x$ cannot run the correction.

Exam note: SCAFFOLD is stochastic controlled averaging. Name the two extra vectors: local control $c_{i}$ and global control $c$ . State that the server sends $x$ and $c$ , and that the client returns differences $Δ y_{i}$ and $Δ c_{i}$ .

The next section makes those arrows visible with three weights and a loss of $12$ .

11.3 Gradients as Directions: A Three-Weight Toy Model

Before the full SCAFFOLD update, we rebuild what a gradient is, using a tiny network so the vectors stay visible.

If a model has only three numbers inside it, can you still see the gradient as three stories, one per weight, instead of as a mysterious blob?

11.3.1 Loss Sensitivity for Three Weights

Assume the whole model has three parameters $w_{1}, w_{2}, w_{3}$ . In the toy, those weights sit at $2$ , $5$ , and $1$ . After a forward pass the loss — the mismatch number we want small — is $12$ . The gradient asks: if I nudge one weight a little, how does that loss move?

Write $\partial L / \partial w_{1}$ for the change in loss $L$ when $w_{1}$ moves. The verbal description is: “if I slightly change each weight, how does my loss change?”

For this toy the three partials are

$\frac{\partial L}{\partial w _{1}} = 4, \frac{\partial L}{\partial w _{2}} = - 2, \frac{\partial L}{\partial w _{3}} = 1$

so the gradient vector is

$g = (4, - 2, 1) .$

If $w_{1}$ rises by a tiny $Δ x$ , loss rises by about $4$ times that nudge. If $w_{2}$ rises a little, loss falls (the partial is negative). If $w_{3}$ rises a little, loss rises only a bit. The first coordinate is the sharp increase. The second is the decrease. The third is a mild increase.

A gradient is the steepest-uphill arrow a hiker feels underfoot. Here the hill lives in three dimensions, one axis per weight. Walking $+ Δ$ along $w_{1}$ is like stepping up a slope of $4$ . Walking $+ Δ$ along $w_{2}$ is walking downhill. The analogy breaks when coordinates mix: a real step moves all three weights at once, so the loss change is the dot product $g \cdot Δ w$ .

Start at $(w_{1}, w_{2}, w_{3}) = (2, 5, 1)$ with loss $L = 12$ . Nudge only the first weight by $0.1$ . The first-order change in loss is $4 \times 0.1 = 0.4$ , so loss would move to about $12.4$ . Nudge only the second weight by $0.1$ . The change is $(- 2) \times 0.1 = - 0.2$ , so loss would move to about $11.8$ . Negative coordinates are not a bug. They are “this weight should grow if we want loss to fall.”

11.3.2 Mini-Batch Gradients on Client 1

Client 1 does not see one gradient forever. Each mini-batch — a small pack of samples — gives a new gradient. Four packs on client 1 produced four different arrows.

The four mini-batch gradients on client 1 are

$g^{(1)} g^{(2)} g^{(3)} g^{(4)} = (4, - 2, 1) = (5, - 1, 2) = (4, - 3, 1) = (6, - 2, 0) .$

The vector changes every iteration. There is still a pattern. The first coordinate hangs around $5$ (values $4, 5, 4, 6$ ). One step below $5$ is $4$ . One step above is $6$ . The second coordinate hangs around $- 2$ or $- 1$ . The third hangs around $1$ , sometimes one step up or down.

The spoken average for client 1 is $c_{1} = (5, - 2, 1)$ . Check the four-pack mean:

$\frac{g ^{(1)} + g ^{(2)} + g ^{(3)} + g ^{(4)}}{4} = (\frac{4 + 5 + 4 + 6}{4}, \frac{- 2 - 1 - 3 - 2}{4}, \frac{1 + 2 + 1 + 0}{4}) = (\frac{19}{4}, \frac{- 8}{4}, \frac{4}{4}) = (4.75, - 2, 1),$

which is “in and around” $(5, - 2, 1)$ . That average direction is client 1’s usual gradient. Gradients are directions where loss rises fast. After many batches, the client’s gradient settles toward this typical arrow.

Plot iteration on the x-axis and the first coordinate of $g$ on the y-axis. The four points sit at $4, 5, 4, 6$ . They bounce, but they bounce around $5$ , not around $50$ . The landmark is that horizontal band near $5$ . The takeaway: noise is real, and so is a stable habit.

11.3.3 Other Clients and the Server Average Direction

Client 2 holds very different data, such as dog samples. Its mini-batch gradients wander, then settle toward a different typical arrow $c_{2}$ . Client 3 settles toward $c_{3}$ . Those usual directions go to the server. The server averages them into a global direction $c$ .

The walkthrough mixed first coordinates $5$ , $- 3$ , and $1$ , then divided by $3$ , giving $1$ . It mixed second coordinates $- 2$ , $4$ , and $0$ , then divided by $3$ , giving about $0.67$ . It mixed third coordinates $1$ , $- 1$ , and $2$ , then divided by $3$ , giving $2/3$ .

Those three spoken coordinate-wise averages recover the missing client vectors as well as $c$ . Client 1 already occupies the first slot of each triple, so

$c_{1} = (5, - 2, 1), c_{2} = (- 3, 4, - 1), c_{3} = (1, 0, 2) .$

Putting the spoken averages together,

$c = (\frac{5 + ( - 3 ) + 1}{3}, \frac{- 2 + 4 + 0}{3}, \frac{1 + ( - 1 ) + 2}{3}) = (1, \frac{2}{3}, \frac{2}{3})$

with $\frac{2}{3} \approx 0.67$ . This $c$ is the average direction preferred by all the clients. It is not client 1’s private habit. It is the group trend.

Shape check: each $c_{i} \in R^{3}$ , so $c \in R^{3}$ . Limiting check: if every client had the same usual arrow, $c$ would equal that arrow. Here the first coordinates $5$ , $- 3$ , and $1$ disagree hard, so the group first-coordinate $1$ is a compromise, not a copy of client 1.

Client 1’s habit $(5, - 2, 1)$ and the group habit $(1, 2/3, 2/3)$ are not the same arrow. That gap is client drift in gradient space. SCAFFOLD will use both arrows in the next section.

11.3.4 Student Questions and Answers

Four different packs can still share one habit. The habit is the average, not one lucky batch.

Q: The four mini-batch gradients on client 1 are all different. Why call $c_{1}$ “the” gradient of that client?

A: Each pack is noisy. The first coordinate still stays near $5$ , not near $50$ or $- 20$ . The usual gradient is the average direction after many packs. That average is the client’s habit, not one lucky batch.

11.3.5 Exam Notes

Do not read a three-number gradient as one mood. Coordinate $4$ means “raising this weight raises loss.” Coordinate $- 2$ means “raising this weight lowers loss.” Do not treat one mini-batch as the client. Do not treat $c_{1}$ and $c$ as the same vector: $c_{1}$ is private, $c$ is the group.

Exam note: A gradient coordinate of $4$ means a small rise in that weight raises loss by about $4$ times the nudge. Negative means loss falls. Be ready to read a three-number gradient as three stories, one per weight, and to average several mini-batch arrows into a usual direction $c_{1}$ .

We now have a current arrow, a habit arrow, and a group arrow. The next section subtracts one and adds the other.

11.4 Control Variates and the SCAFFOLD Correction

Now the current batch can disagree with the client’s habit. SCAFFOLD edits that current gradient before the local step.

If this mini-batch is louder than the client’s long-run habit, should the local walk follow the loud batch, or should it be steered back toward the group?

11.4.1 Current Gradient Versus Usual Gradient

After four packs, take a fifth mini-batch on client 1. The current gradient is $g_{1} = (7, - 1, 3)$ . The client’s usual gradient is still $c_{1} = (5, - 2, 1)$ . The fifth pack is louder than the habit: $7$ instead of about $5$ , and $3$ instead of about $1$ . Mini-batches differ, so a single pack can yank the walk.

The verbal link is: $g_{1}$ is the current gradient for this batch. $c_{1}$ is the average of the mini-batch gradients. $c_{1}$ is the in-built bias, the habitual direction.

Think of $c_{i}$ as memory. Many batches produced many arrows. Their average is the memory of typical direction. In the three-weight toy that memory is an arrow in three-dimensional space. Each coordinate of $c_{i}$ is “how this client usually wants $w_{1}$ , $w_{2}$ , or $w_{3}$ to move.” Each coordinate of $c$ is the same question for the group.

Compare the fifth pack to the habit, coordinate by coordinate:

First: $7$ versus $5$ (this batch pulls $w_{1}$ harder than usual).
Second: $- 1$ versus $- 2$ (this batch is a milder decrease).
Third: $3$ versus $1$ (this batch pulls $w_{3}$ much harder).

If client 1 took a raw step along $g_{1}$ , it would follow a spike, not its own average, and not the group.

11.4.2 Remove Local Drift, Add the Global Average

The correction is: subtract the local habit and add the global habit.

In symbols,

$g_{i}^{corr} = g_{i} - c_{i} + c .$

The verbal description is: “ $g$ minus, remove the bias, and add the global context.” For the fifth pack, start from $g_{1}$ , subtract $c_{1}$ (the average in-built deviation), then add $c$ (the overall average drift). The client’s private tilt comes off. The group’s tilt goes on. The update is steered closer to what helps all clients, not only client 1’s data.

Picture the walk as turning an arrow. Client 1 wants to turn one way because this batch, and because its long-run cat habit, both pull that way. The other clients sit in a cluster. The correction rotates client 1’s step toward that cluster. It does not freeze client 1. It changes the heading.

The three-dimensional arithmetic was not finished out loud in the walkthrough. Completing the vector with the $c$ from the previous section:

$g_{1}^{corr} = (7, - 1, 3) - (5, - 2, 1) + (1, \frac{2}{3}, \frac{2}{3}) = (7 - 5 + 1, - 1 - (- 2) + \frac{2}{3}, 3 - 1 + \frac{2}{3}) = (3, 1 + \frac{2}{3}, 2 + \frac{2}{3}) = (3, \frac{5}{3}, \frac{8}{3}) .$

Sense-check: the first coordinate fell from a spiked $7$ toward the group’s $1$ , and landed at $3$ . The step is still a descent direction, but it is no longer client 1’s private spike. The one-number toy in the next section finishes the same formula with $9 - 5 + 2 = 6$ , which is the numeric form of the same subtract-then-add rule.

The same formula is the one used in the one-number toy in the next section. The three-weight picture is for seeing direction. The one-number picture is for seeing every add and divide.

Standard treatments write the local SGD step as $y_{i} \leftarrow y_{i} - η (g_{i} - c_{i} + c)$ . That is the same correction: the term $(- c_{i} + c)$ is an estimate of how far this client’s usual direction sits from the group’s usual direction.

11.4.3 Why Subtract $c_{i}$ , Not Only Add $c$

A common wrong move is to add the global direction and leave the local habit in place. Then the local drift is still inside the step. $c_{i}$ is average behavior. The current gradient sits on top of that behavior. Subtract the average behavior. What remains is the extra wiggle of this batch. Then add the global behavior so the step faces the group, not the private bias.

If one client has noisy data, or data that is not common across clients, that client can fly off in a private direction. Subtracting $c_{i}$ is the brake on that private flight.

Write the two alternatives side by side. Wrong: $g_{i} + c$ , which still contains the full local habit. Right: $g_{i} - c_{i} + c$ , which replaces the local habit with the group habit and keeps only this batch’s extra wiggle. If $g_{i} = c_{i}$ exactly (this batch matches the habit), the right rule collapses to $c$ : the client steps along the group arrow, not along its private arrow.

Scope: The correction assumes $c_{i}$ is a decent sketch of the client’s mean gradient and $c$ is a decent sketch of the population mean. Early in training those memories are rough. They still get sent and used. They improve as rounds repeat. The formula also assumes $g_{i}$ , $c_{i}$ , and $c$ share the same shape. It does not by itself stop a client from taking many local steps; it rewrites each step’s heading.

Draw three arrows from the same origin in the $(w_{1}, w_{2})$ plane: $g_{1}$ long and cat-tilted, $c_{1}$ along the same tilt but shorter, $c$ pointing toward the group. The corrected arrow is $g_{1}$ minus $c_{1}$ plus $c$ . The landmark is the rotation toward the group, not a freeze at the origin. The takeaway: SCAFFOLD edits direction, not merely step length.

Traps: (1) Adding $c$ without subtracting $c_{i}$ leaves local drift in the step. (2) Treating $c_{i}$ as this batch’s gradient, so you subtract the current $g_{i}$ from itself and get $c$ with no batch information. (3) Mixing coordinates: you must subtract and add vector-wise, not a single scalar. (4) Thinking the client can invent $c$ from local data. It cannot.

11.4.4 Student Questions and Answers

Two questions hit the same formula from two sides: what the symbols mean, and why the minus sign is required.

Q: How are $g_{1}$ and $c_{1}$ related, and what do we do with them?

A: $g_{1}$ is the current mini-batch gradient. $c_{1}$ is the average of the mini-batch gradients, the client’s usual direction. Remove the local drift and add the global average: minus $c_{1}$ plus $c$ . That is the SCAFFOLD correction.

The minus sign is the part students retry. If you only add the group arrow, the private habit is still sitting in the gradient.

Q: Why do we need $g_{i} - c_{i}$ , not a plus on $c_{i}$ ? Why subtract the local control rather than only adding global $c$ ?

A: If we do not remove the local bias and we keep adding, the local drift is not removed. We would add the global trend while the local habit is still sitting in the gradient. $c_{i}$ is average behavior. Subtract that average from the current gradient. The current value was sitting on top of the habit. After the subtraction we have a cleaner gradient, then we add the global direction. If local bias is not removed, the local drift stays inside the step.

11.4.5 Exam Notes

Exam note: Write $g_{i}^{corr} = g_{i} - c_{i} + c$ and say in words: remove local bias, add global context. A short “why subtract?” item is likely, because that is the confusion that needed a second pass. Be ready to finish the fifth-batch vector $(7, - 1, 3) - (5, - 2, 1) + (1, 2/3, 2/3)$ .

The three-weight picture taught heading. The next toy collapses the model to one number so every add and divide is visible.

11.5 A Full SCAFFOLD Round With Two Clients

The three-weight picture taught direction. The next toy collapses the model to one number so every add and divide is visible. The same logic extends component-wise to real vectors.

This section is a procedure, not a new definition. We send, train, recompute controls, and upload differences.

If the whole network is replaced by a single number $10$ , can you still watch SCAFFOLD’s minus-then-plus rule, the two local hops, and the two uploaded pairs without losing a digit?

11.5.1 What the Server Sends

Purpose. Run one communication round so two clients start from the same global model, correct their local steps, and return changes the server can average.

Inputs. Global weights $x = 10$ . Global control $c = 2$ . Client 1’s stored control $c_{1} = 5$ . Client 2’s stored control $c_{2} = 1$ . Local learning rate $η = 0.1$ . Local steps $K = 2$ .

Outputs. From each client $i$ , a pair $(Δ y_{i}, Δ c_{i})$ with $Δ y_{i} = y_{i} - x$ and $Δ c_{i} = c_{i}^{+} - c_{i}$ .

Start of round: global weights $x = 10$ . Global control $c = 2$ . The verbal description of $c$ is the biases of various clients averaged together; early on that number may not mean much yet, but the server still sends it. Both client 1 and client 2 receive $x$ and $c$ . Client 1 already holds old local control $c_{1} = 5$ . Client 2 holds $c_{2} = 1$ .

Client 1 copies the model into a local weight $y_{1}$ , starting at $y_{1} = x = 10$ .

11.5.2 Client 1: Corrected Steps and New Control

On a new batch, the raw gradient is $g_{1} = 9$ . That $9$ already contains the in-built bias $5$ . Subtract the local habit and add the global trend:

$g_{1}^{corr} = 9 - 5 + 2 = 6.$

The verbal description is: “I am not allowing, instead of $9$ , now I make it $6$ .” The client is not allowed to deviate as hard as the raw batch wants.

Steps on a client. (1) Copy $y_{i} \leftarrow x$ . (2) For each local step, form $g_{i}^{corr} = g_{i} - c_{i} + c$ and walk $y_{i} \leftarrow y_{i} - η g_{i}^{corr}$ . (3) Recompute $c_{i}^{+}$ from how far the weights moved. (4) Upload differences, not raw new values.

Local training then runs two epochs. Weights move $10 \to 9.4$ and then $9.4 \to 8.8$ . Each drop is $0.6$ . If the corrected gradient stays $6$ , a learning rate $η = 0.1$ gives a step $η \cdot 6 = 0.6$ , which matches those two hops:

$y_{1} y_{1} \leftarrow 10 - 0.1 \times 6 = 9.4, \leftarrow 9.4 - 0.1 \times 6 = 8.8.$

The local model after training is $y_{1} = 8.8$ .

Next, recompute the client control. The high-level mix is: old $c_{1}$ minus current global $c$ , plus a corrected-gradient term recovered from how far the weights moved. The two matching expressions used in the walkthrough are

$c_{1}^{+} = c_{1} - c + 6 = 5 - 2 + 6 = 9$

and

$\frac{x - y _{1}}{0.2} = \frac{10 - 8.8}{0.2} = \frac{1.2}{0.2} = 6.$

The $0.2$ is the product of local-step count and learning rate, not a count of clients. With two local steps and $η = 0.1$ ,

$K η = 2 \times 0.1 = 0.2.$

The general form is

$c_{i}^{+} = c_{i} - c + \frac{x - y _{i}}{K η} .$

Plug in client 1:

$c_{1}^{+} = 5 - 2 + \frac{10 - 8.8}{2 \times 0.1} = 3 + 6 = 9.$

The walkthrough mixed extra words around that $0.2$ divisor. The arithmetic that was actually computed is $(10 - 8.8) /0.2 = 6$ , and $N$ (the fleet size) does not belong in this client-side divisor. Standard Option II of SCAFFOLD is this same formula. (Option I would recompute a full local gradient at $x$ ; the numeric round uses Option II.)

Because both local hops used the same corrected gradient $6$ ,

$y_{1} = x - K η \cdot 6 ⟹ \frac{x - y _{1}}{K η} = 6,$

and then $c_{1}^{+} = c_{1} - c + 6 = 9$ , which equals the raw gradient $g_{1}$ . In this constant-gradient toy, the new control stores the uncorrected local direction.

The client does not upload $y_{1} = 8.8$ and $c_{1}^{+} = 9$ as raw values. It uploads differences. The model change is $8.8 - 10 = - 1.2$ (spoken once as “became 8,” but the difference used is $- 1.2$ , which matches $8.8$ ). The control change is $9 - 5 = 4$ . Client 1 sends

$(Δ y_{1}, Δ c_{1}) = (- 1.2, 4) .$

Sense-check: the model fell by $1.2$ over two steps of $0.6$ . The control rose by $4$ , which equals $- c + 6 = - 2 + 6$ .

11.5.3 Client 2: The Same Recipe With Different Habit

Client 2 starts from the same $x = 10$ and the same global $c = 2$ , but its local control is $c_{2} = 1$ . Raw gradient $g_{2} = 3$ . Corrected gradient:

$g_{2}^{corr} = 3 - 1 + 2 = 4.$

Weights move $10 \to 9.6$ then to $9.2$ . Each hop of $0.4$ matches $η \cdot 4 = 0.4$ when $η = 0.1$ :

$y_{2} y_{2} \leftarrow 10 - 0.1 \times 4 = 9.6, \leftarrow 9.6 - 0.1 \times 4 = 9.2.$

Final local weight $y_{2} = 9.2$ . New control:

$c_{2}^{+} = 1 - 2 + \frac{10 - 9.2}{0.2} = - 1 + 4 = 3.$

Model difference: $9.2 - 10 = - 0.8$ . Control difference: $3 - 1 = 2$ . Client 2 sends

$(Δ y_{2}, Δ c_{2}) = (- 0.8, 2) .$

In words one pass said: “You sent me a model. I changed the model by $- 0.8$ . I changed the $c$ value by $1$ .” Wait: $3 - 1 = 2$ , not $1$ . The spoken “changed the $c$ value by $1$ ” does not match $c_{2}^{+} - c_{2} = 2$ . Keep both lines in view, and prefer the send pair $(- 0.8, 2)$ , which is the pair that continues into the server average.

Every participating client sends $Δ y_{i} = y_{i} - x$ and $Δ c_{i} = c_{i}^{+} - c_{i}$ .

Time cost: each client does $K$ local gradient steps plus two vector subtractions for the deltas. Communication cost: two model-shaped payloads instead of one (the extra $Δ c_{i}$ ). On a tiny device that extra vector is not free. The one-number toy hides that cost; a million-weight model does not.

When to use this round. Use the corrected local walk when clients are heterogeneous and you can afford extra control state. If you cannot store or ship a second model-sized vector, the next section’s FedProx leash is the alternative. Do not mix the two recipes in one round.

11.5.4 Student Questions and Answers

The server asked for the change, not the new snapshot. That is why the upload is a pair of differences.

Q: After local training, why send differences such as $- 1.2$ and $4$ , not the new weight $8.8$ and the new control $9$ ?

A: The server asked for the change. Client 1 received $10$ and ended at $8.8$ , so the model moved by $- 1.2$ . The control moved from $5$ to $9$ , so the control moved by $4$ . The server will average those changes across the clients that took part.

Exam note: Reproduce the one-dimensional round: $x = 10$ , $c = 2$ , $c_{1} = 5$ , $g = 9$ , corrected $6$ , local path $10 \to 9.4 \to 8.8$ , $c_{1}^{+} = 9$ , send $(- 1.2, 4)$ ; client 2 path to $9.2$ , send $(- 0.8, 2)$ . Write $c_{i}^{+} = c_{i} - c + (x - y_{i}) / (K η)$ .

The clients have sent their pairs. The server now has to mix them, including the case where only some of the fleet showed up.

11.6 Server Aggregation and Partial Participation

Once every chosen client has sent $(Δ y_{i}, Δ c_{i})$ , the server updates $x$ and $c$ .

The clients did the walking. What exact average should the server apply, and what happens when only some of the fleet joined the round?

11.6.1 Averaging the Model Changes

Two clients took part, so the subset size is $∣ S ∣ = 2$ . The two model deltas are $- 1.2$ and $- 0.8$ . Their mean is

$Δ x = \frac{- 1.2 + ( - 0.8 )}{2} = \frac{- 2}{2} = - 1.$

A first pass said “minus 2 by 2, which is nothing but 1.” That dropped the sign. The corrected pass is: on average the selected clients recommend decreasing the global model by $1$ . Then

$x^{+} = x + Δ x = 10 + (- 1) = 9.$

The new global model is $9$ . A server learning rate $η_{g}$ can scale $Δ x$ if we want a gentler or sharper apply:

$x^{+} = x + η_{g} Δ x, Δ x = \frac{1}{∣ S ∣} i \in S \sum Δ y_{i} .$

In this numeric walkthrough the scale is $η_{g} = 1$ , so $x$ moves from $10$ to $9$ . Standard SCAFFOLD uses this same mean of $(y_{i} - x)$ .

Sign-check: both clients lowered their local copy ( $8.8$ and $9.2$ are both below $10$ ), so the global model should fall, not rise. The value $9$ sits between $8.8$ and $9.2$ , which is where an unweighted mean of the two finished local models would sit.

11.6.2 Averaging the Control Changes, Then Scaling by Who Showed Up

The two control deltas are $4$ and $2$ . Their mean is $3$ . A naive apply would be $c \leftarrow 2 + 3 = 5$ . The method does not always add the raw mean. Only some clients participate. Here $∣ S ∣ = 2$ took part and $N = 4$ clients exist in the system. Scale the control step by the participation ratio $∣ S ∣/ N = 2/4$ :

$c^{+} = c + \frac{∣ S ∣}{N} \cdot mean_{i \in S} (Δ c_{i}) = c + \frac{1}{N} i \in S \sum Δ c_{i} .$

With the toy numbers,

$c^{+} = 2 + \frac{2}{4} \cdot 3 = 2 + 1.5 = 3.5.$

The verbal line that sounded like “0.53, which is nothing but 3.5” is the same $0.5 \times 3$ path with a slurred “0.5 times 3.” Arithmetic that matches is $2 + (2/4) \times 3 = 3.5$ .

So the next broadcast is a new model $x = 9$ and a new global direction. If the naive unscaled add had been used, that direction would have been $5$ . With partial participation it is $3.5$ .

Why divide by $N$ rather than by $∣ S ∣$ ? Global $c$ is meant to track the fleet average of the $c_{i}$ values, including clients who sat out and did not change. Adding the subset mean as if it were the whole fleet would pretend two updates were four updates. Multiplying by $∣ S ∣/ N$ keeps the missing clients’ old controls in the average.

If all four clients had joined, $∣ S ∣/ N = 1$ and the scale would disappear: $c^{+} = 2 + 3 = 5$ . Partial participation is the only reason the toy lands on $3.5$ .

11.6.3 Intuition of the Whole Loop

The server stores global weights and a global average drift. Each client starts from the global model, trains on local data, forms a local bias picture $c_{i}$ , and corrects the local step by subtracting that bias and adding the global control. The client sends the difference in weights and the difference in $c$ . The server averages the weight changes into a new global model and averages the $c$ changes into a new global control, scaled by how many clients joined. Repeat. Clients are pulled toward the global weights instead of toward their private skewed data. The loop converges toward those global weights rather than toward each client’s biased slice.

Draw a timeline. Round $t$ : server ships $(x, c)$ . Each chosen client walks, then ships $(Δ y_{i}, Δ c_{i})$ . Server writes $x^{+}$ and $c^{+}$ . Round $t + 1$ repeats. The landmark on the number line is $x : 10 \to 9$ and $c : 2 \to 3.5$ . The takeaway: two different averages, two different scalings.

Scope: The $∣ S ∣/ N$ factor assumes $c$ is a fleet-wide average and that idle clients keep their old $c_{i}$ . It is not a learning-rate knob like $η$ . If $∣ S ∣ = N$ , the factor is $1$ . If $∣ S ∣ = 0$ , there is no round. The model update does not use $∣ S ∣/ N$ in the toy; only the control update does.

On a number line, mark $x = 10$ and the two finished local points $8.8$ and $9.2$ . The mean delta $- 1$ lands at $9$ , between those two points. Mark $c = 2$ and the naive add at $5$ ; the scaled update lands at $3.5$ , closer to the old $c$ because half the fleet stayed home. The landmark is that split: model mean versus participation-scaled control mean.

Traps: (1) Dropping the minus sign so $Δ x$ becomes $+ 1$ and $x$ rises to $11$ . (2) Adding the mean $Δ c$ with no $∣ S ∣/ N$ factor, landing on $5$ instead of $3.5$ . (3) Scaling the model update by $∣ S ∣/ N$ when the walkthrough scaled only $c$ . (4) Averaging finished weights $8.8$ and $9.2$ while forgetting that the uploaded objects were deltas.

11.6.4 Student Questions and Answers

“Just averaging” is too vague. Name which vectors are averaged and which factor multiplies the control step.

Q: Once every client has sent its pair, what is the server step? Is it just averaging?

A: It is not “averaging” as a vague slogan. The server averages the received $Δ y_{i}$ values. With $- 1.2$ and $- 0.8$ that mean is $- 1$ , so the global $x$ moves from $10$ to $9$ . It also updates $c$ . $S$ is the participating set. $N$ is the total number of clients. The control update uses a factor $∣ S ∣/ N$ , not a blind add of the mean $Δ c$ . Average delta $y$ to move $x$ , and update $c$ with factor $S$ over $N$ .

The short story of the whole loop is the exam-ready intuition.

Q: What is the intuition of SCAFFOLD, as a short story of server and clients?

A: The server manages global weights and a global average drift. Each client starts from the global model and trains on local data. It forms its local bias $c_{i}$ . It corrects the local walk by subtracting that bias and adding the global control. It sends the difference of its weights and of its $c$ back. The server averages the weight changes into a new global model and averages the received $c$ changes to update global $c$ . The loop keeps running so training converges toward the global weights rather than toward each client’s biased data.

11.6.5 Exam Notes

Exam note: Be able to compute $Δ x$ from a short list of $Δ y_{i}$ , then $x^{+} = x + Δ x$ . Be able to apply $c^{+} = c + (∣ S ∣/ N) \times mean (Δ c_{i})$ when only a subset joins. Mention that a server learning rate may scale the $x$ step. Keep the sign: mean $- 1$ means decrease, not increase.

SCAFFOLD needed extra arrows. FedProx will try the same drift pain with a rubber band and no extra vectors.

11.7 FedProx: A Rubber-Band Penalty on Distance

After SCAFFOLD, the second method is FedProx. It also fights drift, but it does not keep control vectors. It penalizes distance from the global model.

If extra control arrows are too expensive to store and ship, can you still stop a client from sprinting away — using only the weights you already have?

11.7.1 Rubber-Band Picture

If some weights run one way and some run another way, a plain average can hurt the global model. We want local walks to stay near the global trend. Think of a rubber band tied between the global model and the local model. The farther the local model walks away, the harder the band pulls back. The local model is not free to sprint. It stays closer to the global point.

The strength of that band is a constant $μ$ (spoken “mu”). $μ$ is the proximal weight: how hard we leash the local model to the global model. You choose it. It is a scalar, not a vector.

A speed limiter on a car is a cousin of this idea: the engine can still request a fast acceleration, but the limiter caps how far you go. The analogy breaks because FedProx does not cap speed with a hard wall. It adds a smooth pull that grows with distance.

11.7.2 Penalty Gradient From a Quadratic Term

The local goal is not only “fit my data.” It is “fit my data, but pay for walking away from the global weights.” Let $y$ be the local weights and $x$ the global weights. The extra term is a quadratic in the gap.

Write the penalty as

$P (y) = \frac{μ}{2} (y - x)^{2}$

in the one-number toy, or $\frac{μ}{2} ∥ y - x ∥^{2}$ for vectors. Differentiate with respect to $y$ . The verbal derivative is: $μ /2$ times $2$ times $y - x$ ; the $2$ and the $1/2$ cancel, leaving

$\frac{d P}{d y} = \frac{μ}{2} \cdot 2 (y - x) = μ (y - x) .$

That is the penalty gradient. FedProx adds it to the data gradient $g$ . The local step becomes

$y \leftarrow y - η (g + μ (y - x)) .$

Because $y - x$ is often negative when $y$ has already dropped below $x$ , the penalty gradient is negative and shrinks the step that would have run farther away.

If $μ = 0$ , the extra term is zero and the method collapses to federated averaging.

Shape check: $y$ and $x$ have the same shape, so $y - x$ matches $g$ . Domain check: $∥ y - x ∥^{2}$ is never negative, and $μ \geq 0$ is the usual choice. Limiting check: $μ = 0$ recovers $y \leftarrow y - η g$ .

11.7.3 Numeric Walkthrough With $μ = 2$ and $η = 0.1$

Global weight $x = 10$ . Local weight starts at $y = 10$ . Gap is $0$ , so penalty is $0$ .

First local update uses data gradient $g = 9$ and learning rate $η = 0.1$ :

$y \leftarrow y - η g = 10 - 0.1 \times 9 = 9.1.$

The local model has begun to move away. The gap is $10 - 9.1 = 0.9$ . That is “orange, not danger”: close, but no longer glued.

Without a penalty, a second gradient of $9$ would do $9.1 - 0.1 \times 9 = 8.2$ . FedProx says that jump from $9.1$ to $8.2$ is too far.

Now form the penalty at $y = 9.1$ with working value $μ = 2$ (a first attempt used $μ = 0.1$ , then corrected to $μ = 2$ ):

$μ (y - x) = 2 \times (9.1 - 10) = 2 \times (- 0.9) = - 1.8.$

Modified gradient:

$g + μ (y - x) = 9 + (- 1.8) = 7.2.$

Use $7.2$ instead of $9$ . From $y = 9.1$ ,

$y \leftarrow 9.1 - 0.1 \times 7.2 = 9.1 - 0.72 = 8.38.$

Without FedProx the second stop was $8.2$ , distance $1.8$ from the global $10$ . With FedProx the stop is $8.38$ , distance $1.62$ . The band did not reverse the walk. It shortened it.

A side path applied $7.2$ from the original $10$ , which would give $10 - 0.72 = 9.28$ . That path uses the wrong starting point. The $8.38$ versus $8.2$ pair is the one used to compare distance, and it uses the second-step starting point $9.1$ .

Without FedProx the update is $y \leftarrow y - η g$ . With FedProx it is $y \leftarrow y - η (g + μ (y - x))$ . Because $μ (y - x)$ is negative in this walk, that is the same as shrinking the gradient, spoken as “minus penalty.”

Sense-check: $8.38 > 8.2$ , so the leashed client stayed nearer $10$ . The band pulled back by $0.18$ of distance, not by reversing the sign of the step.

Plot local weight on the y-axis and local step index on the x-axis. Unleashed path: $10 \to 9.1 \to 8.2$ . Leashed path: $10 \to 9.1 \to 8.38$ . The landmark is the second hop. The takeaway: same first hop (gap was zero), shorter second hop (band is taut).

11.7.4 Local Objective and the Role of $μ$

Let device $k$ have local fit $F_{k} (w)$ . FedAvg would minimize that fit alone. FedProx adds a penalty that grows with distance from the global model $w^{t}$ :

$w min (F_{k} (w) + \frac{μ}{2} ∥ w - w^{t} ∥^{2}) .$

Here $w^{t}$ is the global model and $w$ is the client model. As distance grows, penalty grows, and the effective gradient shrinks, so the client does not run as far. Standard write-ups of FedProx use this same proximal term. Because the extra term is a function of $w$ , local training needs a gradient-based method to add $μ (w - w^{t})$ to the data gradient.

If $μ$ goes from $2$ to $50$ , the band gets stiff. A large gap is multiplied by a large $μ$ , so the subtractive penalty is huge, and the local model stays closer to $w^{t}$ . A first guess was “updates become slower.” The correction is: not slower as the main story. The model becomes closer to the global $w^{t}$ . At $μ = 0$ it is ordinary FedAvg.

Procedure each round: server sends the global model; client computes a data gradient; FedProx adds the penalty; the smaller effective step keeps weights nearer the global model; then the server averages.

Scope: FedProx applies when local training is gradient-based and you can evaluate $y - x$ each step. It does not store control variates. A huge $μ$ can freeze the client near $w^{t}$ and starve local learning. A tiny $μ$ is almost FedAvg. The method also does not, by itself, fix adversarial clients; it only leashes distance.

11.7.5 Student Questions and Answers

$μ$ is the one knob. Large $μ$ pulls harder.

Q: Any doubt on FedProx? What does $μ$ do?

A: $μ$ decides the strength of the rubber band. You choose it. Large $μ$ pulls harder toward the global model.

The follow-up is the trap: people hear “bigger penalty” and say “slower updates.” The intended reading is “closer to $w^{t}$ .”

Q: If $μ$ rises from $2$ to $50$ , what is the impact?

A: Highest penalty. If the gap is large, you subtract a bigger number, so the effective gradient shrinks. A first reading was that updates become slower. The intended reading is that the local model becomes closer to the global model $w^{t}$ , not merely slower. If $μ = 0$ , the method is normal federated averaging.

Traps: (1) Applying the modified gradient $7.2$ from $10$ instead of from $9.1$ . (2) Leaving $μ = 0.1$ after the correction to $μ = 2$ . (3) Hearing large $μ$ as only “slower” instead of “closer to $w^{t}$ .” (4) Forgetting that $μ = 0$ is FedAvg.

11.7.6 Industry Applications and Exam Notes

Latest aggregation papers keep finding a new failure mode and shipping a new mix rule. Thesis work can survey those mix rules and try a new one. FedProx is the distance-leash member of that family. Cross-silo hospitals that cannot store a second model-sized control vector still get a leash: one scalar $μ$ on each site. Keyboard models on phones can use the same trick when uplink bytes are scarce.

This sits next to SCAFFOLD as the other standard answer to client drift: change the local objective, not the local direction memory.

Exam note: Derive the penalty gradient $μ (y - x)$ from $\frac{μ}{2} ∥ y - x ∥^{2}$ . Run the $10 \to 9.1 \to 8.2$ versus $8.38$ numbers. State the $μ = 0$ FedAvg limit and the large- $μ$ “stay close to $w^{t}$ ” limit. The large- $μ$ story is “closer,” not merely “slower.”

Both methods fight drift. They do not fight it the same way. The next section names the disagreements one by one.

11.8 SCAFFOLD Versus FedProx

Both methods are averaging methods that fight client drift. They agree that a wild local walk can poison the global model. They disagree about what they constrain and what extra state they store. A fair comparison has to name those disagreements one by one, because an exam answer that says only “both reduce drift” is too thin.

If two tools both claim to stop drift, how do you tell them apart on an exam without a single vague sentence?

11.8.1 Control Strength

In SCAFFOLD the correction is always of the form minus $c_{i}$ plus $c$ . There is no knob that says “correct only halfway.” The size of the correction is whatever $c_{i}$ and $c$ currently are.

SCAFFOLD always uses

$g_{i}^{corr} = g_{i} - c_{i} + c$

with no $μ$ -style multiplier in front of the correction. FedProx instead uses

$g^{prox} = g + μ (y - x)$

and $μ$ is that knob. You choose how strong the rubber band is. That is the first practical difference: FedProx lets you tune the leash. SCAFFOLD’s leash is the control vectors themselves.

11.8.2 Extra Vectors Versus On-the-Fly Distance

SCAFFOLD stores two extra control vectors: local $c_{i}$ on the client and global $c$ on the server. That is extra memory. Those vectors also travel, so there is extra communication. FedProx does not keep those controls. It uses the model weights it already has. Distance $y - x$ is computed on the fly and used to shrink the gradient. The only extra scalar is $μ$ .

If a device is tiny, extra vectors the size of the model are not free. If the uplink is tiny, sending $Δ c_{i}$ beside $Δ y_{i}$ is not free. FedProx’s extra cost is mostly the extra math of the penalty, not extra payloads.

Side-by-side on the dimensions that actually differ:

Dimension	SCAFFOLD	FedProx
What is constrained	Gradient direction	Distance from global weights
Strength knob	None; always $- c_{i} + c$	Scalar $μ$
Extra state	Local $c_{i}$ and global $c$	None besides $μ$
Extra communication	$Δ c_{i}$ with $Δ y_{i}$	No extra vector
Global $c$ needed on the client	Yes	No
$μ = 0$ limit	Not a $μ$ method	Recovers FedAvg
Local change	Rewrite the gradient	Add a proximal penalty to the objective

When to pick which: pick SCAFFOLD when you can store and ship control vectors and you care about aligning headings. Pick FedProx when you want one scalar leash and no extra model-sized payloads.

11.8.3 Gradient Direction Versus Distance

FedProx says: do not move too far from the global model. That is distance based (proximal regularization). SCAFFOLD says: gradients should move in the same direction. That is gradient based. SCAFFOLD subtracts the client’s usual direction and adds the global usual direction, so the arrow itself is rewritten. FedProx leaves the data gradient’s story in place and adds a penalty that limits how far weights may travel.

SCAFFOLD’s client also depends on the global $c$ . FedProx’s client does not receive a global control vector. Its leash is the received global weights $x$ (or $w^{t}$ ) plus $μ$ . In that sense FedProx is “completely local” once $x$ has arrived: the client never asks “what is the group’s usual gradient?” It only asks “how far am I from the global weights?”

A short pair of slogans helps:

FedProx: limit movement.
SCAFFOLD: correct direction.

Limiting movement can still leave a biased heading, just a shorter step along that heading. Correcting direction can still take a long step, but the heading is the group heading. Those are not the same fix.

11.8.4 Objective Change Versus Gradient Rewrite

FedProx changes the objective: local loss plus a proximal penalty. SCAFFOLD changes the gradient by bias correction. One is a penalty on position. The other is a correction of direction. Which one wins on a given dataset is not something to claim from the toy numbers alone. That needs larger data runs. Until those runs exist, treat the two as different tools for the same family of drift problems, not as ranked winners.

Scope: The toy numbers show how each method edits a step. They do not rank test accuracy. Heterogeneous photo clients, hospital sites, and keyboard models can prefer different mix rules. A thesis that ships a new aggregator should name the failure mode first, then pick a constraint (direction versus distance), then measure on large data.

11.8.5 Student Questions and Answers

The comparison question is one bundle: similar goal, different constraint, different state.

Q: What is similar, and what is different, between SCAFFOLD and FedProx?

A: Both are averaging methods that try to limit harmful local walks. A major difference: SCAFFOLD has no $μ$ -style knob on correction strength; it always subtracts $c_{i}$ and adds global $c$ . FedProx uses $μ$ to set how hard to pull back. SCAFFOLD must store two extra $c$ vectors, local and global. FedProx operates on the existing weights plus $μ$ . FedProx is distance based: do not move too far from the global model. SCAFFOLD is gradient based: make directions agree, with extra controls. SCAFFOLD needs extra memory and extra communication. FedProx computes the gap on the fly. FedProx modifies the objective with a proximal penalty. SCAFFOLD modifies the gradient with bias correction. Do not claim a winner until you run large data.

11.8.6 Exam Notes

Traps: (1) Stopping at “both reduce drift.” (2) Giving FedProx a control vector it does not have. (3) Giving SCAFFOLD a $μ$ knob it does not have. (4) Ranking the two from the one-number toys.

Exam note: A comparison table is high value: knob $μ$ versus no knob; extra controls versus none; extra memory and communication versus none; direction correction versus distance penalty; global $c$ needed versus not needed. Also: $μ = 0$ recovers FedAvg.

Exam Guidance Summary

Expect questions that mix a short numeric toy with a “why” sentence.

Write FedAvg as a sample-count weighted mix of client weights. Say why non-IID client data (cats versus dogs versus elephants) makes that mix unsafe.
Define SCAFFOLD as stochastic controlled averaging. Name $c_{i}$ as client memory of typical gradient direction and $c$ as the group’s typical direction.
Write $g^{corr} = g - c_{i} + c$ and explain subtract-then-add. A “why not only add $c$ ?” probe is likely.
Reproduce the one-dimensional round: $x = 10$ , $c = 2$ , $c_{1} = 5$ , $g = 9$ , corrected $6$ , local path $10 \to 9.4 \to 8.8$ , $c_{1}^{+} = 9$ , send $(- 1.2, 4)$ ; client 2 path to $9.2$ , send $(- 0.8, 2)$ ; $Δ x = - 1$ , $x^{+} = 9$ ; control scale $∣ S ∣/ N$ with $2/4$ giving $c^{+} = 3.5$ rather than a blind $5$ .
FedProx: rubber-band story, penalty $μ (y - x)$ , numbers $9.1$ , $8.2$ versus $8.38$ , distances $1.8$ versus $1.62$ .
Limits: $μ = 0$ is FedAvg; large $μ$ keeps $w$ close to $w^{t}$ . The large- $μ$ story is “closer,” not merely “slower.”
Comparison: SCAFFOLD corrects direction with extra controls; FedProx limits distance with $μ$ and no extra vectors. A table on knob, memory, communication, and heading versus distance is high value.
Thesis path: survey latest aggregation methods and try a new mix rule for a new failure mode.

Exam note: Numeric toys will be small enough to finish by hand. Show every sign. Name the method’s extra state (controls versus $μ$ ) in the same answer as the arithmetic.

Key Industry Applications

On-device photo clients that each see one animal class are the teaching stand-in for any phone fleet whose label mix is skewed. A cat-heavy camera roll and a dog-heavy camera roll are the same non-IID pattern as a sports-slang keyboard versus a medical-term keyboard.

Cross-site medical or phone data that is non-IID is the setting where FedAvg’s plain weighted mix is not enough. One hospital’s disease mix is not another hospital’s mix, and the raw records stay on site.

Production federated systems must pick a mix rule. FedAvg is the simple default. SCAFFOLD adds control variates and extra communication to align gradient directions when devices can store and ship a second model-sized vector. FedProx adds a proximal leash when you want a single scalar $μ$ and no extra vectors — useful on tight uplinks and tiny memory.

Research and thesis work still mint new aggregators (FedDSS was named as another member of that family) because each paper isolates one failure mode and writes a method for it. The industry move is the same as the research move: name the failure, pick a constraint, then measure on real client skew rather than on the one-number toy alone.

DML Lecture 11 notes · SCAFFOLD and FedProx: Controlling Client Drift

Distributed Machine Learning· postgraduate· 2026-09-11

Sections Breakdown

1Combining Client Models When Data Is Not Shared

The server mixes client weights into one global model. Non-IID slices (cats, dogs, elephants) create client drift, so FedAvg's sample-count weighted average can pull the shared model off course.

2SCAFFOLD: Stochastic Controlled Averaging

SCAFFOLD is stochastic controlled averaging: extra control variates remember each client's usual gradient and the group's usual gradient so local steps can be corrected during training, not only after averaging.

3Gradients as Directions: A Three-Weight Toy Model

A three-weight toy turns the gradient into three loss-sensitivity stories. Client 1's mini-batch packs average near (5, -2, 1); the server mixes client habits into global c = (1, 2/3, 2/3).

4Control Variates and the SCAFFOLD Correction

SCAFFOLD rewrites the current mini-batch gradient as g_i minus local c_i plus global c, so a spiked fifth batch is rotated toward the other clients instead of following private drift.

5A Full SCAFFOLD Round With Two Clients

A one-number round traces client 1 from x=10 to y=8.8 with corrected gradient 6 and send pair (-1.2, 4), and client 2 to 9.2 with send pair (-0.8, 2).

6Server Aggregation and Partial Participation

The server averages model deltas to move x from 10 to 9, then scales the mean control delta by |S|/N = 2/4 so c moves from 2 to 3.5 rather than a blind 5.

7FedProx: A Rubber-Band Penalty on Distance

FedProx leashes local training with a proximal penalty whose gradient is mu times (y minus x). With mu=2 the second step lands at 8.38 instead of 8.2; large mu means closer to the global model, not merely slower.

8SCAFFOLD Versus FedProx

Both methods fight client drift by averaging, but SCAFFOLD rewrites gradient direction with extra control vectors and no mu knob, while FedProx limits distance with a proximal penalty and no extra vectors.

9Exam Guidance Summary

Exam items mix a short numeric toy with a why sentence: FedAvg weights, SCAFFOLD correction and one-dimensional round, FedProx rubber-band numbers, and a comparison table.

10Key Industry Applications

Photo, keyboard, and hospital fleets with skewed labels need a mix rule beyond FedAvg: SCAFFOLD when extra controls are affordable, FedProx when only a mu leash fits the device.

Postgraduate students in Distributed Machine Learning

Exam Revision Notes

Below is the distilled, exam-ready core. Every entry comes from the full explanation above. Use this section for rapid review; return to the main notes when a point needs more context.

Combining Client Models When Data Is Not Shared

Must-know: FedAvg mixes client weights as a sample-count weighted average; non-IID client data can make that mix unsafe.

$w \leftarrow i = 1 \sum m \frac{n _{i}}{N} w_{i}, N = i = 1 \sum m n_{i} .$

⚠️ Top pitfall: Treating FedAvg as an unweighted mean, or assuming extra local steps always help.

Self-check: Three clients with n=(100,50,50) and w=(1,3,5): what is the FedAvg mix?

Connects to: SCAFFOLD: Stochastic Controlled Averaging, FedProx: A Rubber-Band Penalty on Distance

SCAFFOLD: Stochastic Controlled Averaging

Must-know: SCAFFOLD sends both x and c to the client and gets back Delta y_i and Delta c_i; there is no peer-to-peer channel.

$y_{i} \leftarrow y_{i} - η (g_{i} - c_{i} + c)$

⚠️ Top pitfall: Sending only the model x, or confusing the current gradient g_i with the stored control c_i.

Self-check: Why can a client not compute global c from local data alone?

Connects to: Gradients as Directions: A Three-Weight Toy Model, Control Variates and the SCAFFOLD Correction

Gradients as Directions: A Three-Weight Toy Model

Must-know: A gradient coordinate is how loss moves when that weight is nudged; a client's usual gradient is the average over noisy mini-batches.

$c = (1, 2/3, 2/3)$

⚠️ Top pitfall: Calling one noisy mini-batch 'the' client gradient, or mixing up local c_1 with global c.

Self-check: Average the four client-1 packs. How close is the mean to (5, -2, 1)?

Connects to: Control Variates and the SCAFFOLD Correction

Control Variates and the SCAFFOLD Correction

Must-know: Corrected gradient is current g minus local habit c_i plus global habit c; skipping the minus leaves local drift in the step.

$g_{i}^{corr} = g_{i} - c_{i} + c$

⚠️ Top pitfall: Adding global c without subtracting local c_i.

Self-check: Why is g_i + c the wrong correction?

Connects to: Gradients as Directions: A Three-Weight Toy Model, A Full SCAFFOLD Round With Two Clients

A Full SCAFFOLD Round With Two Clients

Must-know: Local control updates as c_i minus c plus (x minus y_i) over K eta; clients upload differences, not raw new weights.

$c_{i}^{+} = c_{i} - c + \frac{x - y _{i}}{K η}$

⚠️ Top pitfall: Uploading y_i and c_i^{+} instead of deltas, or putting fleet size N in the 0.2 client-side divisor.

Self-check: Client 1: 9 - 5 + 2 equals what, and what pair is sent?

Connects to: Control Variates and the SCAFFOLD Correction, Server Aggregation and Partial Participation

Server Aggregation and Partial Participation

Must-know: x uses the mean model delta; c uses that mean times |S|/N when participation is partial.

$c^{+} = c + \frac{∣ S ∣}{N} \cdot mean (Δ c_{i})$

⚠️ Top pitfall: Dropping the minus sign of Delta x, or adding the raw mean Delta c as if the whole fleet had joined.

Self-check: If |S|=2, N=4, mean Delta c=3, and c=2, what is c^+?

Connects to: A Full SCAFFOLD Round With Two Clients, FedProx: A Rubber-Band Penalty on Distance

FedProx: A Rubber-Band Penalty on Distance

Must-know: Penalty gradient is mu(y-x); mu=0 is FedAvg; large mu keeps w close to w^t rather than merely slowing updates.

$w min (F_{k} (w) + \frac{μ}{2} ∥ w - w^{t} ∥^{2})$

⚠️ Top pitfall: Reading large mu as slower rather than closer, or applying the modified gradient from the wrong starting y.

Self-check: From y=9.1, x=10, mu=2, g=9, eta=0.1, what is the next y?

Connects to: Combining Client Models When Data Is Not Shared, SCAFFOLD Versus FedProx

SCAFFOLD Versus FedProx

Must-know: SCAFFOLD is gradient-based direction correction with extra controls; FedProx is distance-based with a mu knob and no extra vectors.

$g_{i}^{corr} = g_{i} - c_{i} + c$

⚠️ Top pitfall: Saying only that both reduce drift, or ranking the two from the toy numbers.

Self-check: Which method needs global c on the client, and which recovers FedAvg at mu=0?

Connects to: Control Variates and the SCAFFOLD Correction, FedProx: A Rubber-Band Penalty on Distance

Previous LectureLecture 10 · Federated Averaging and SCAFFOLD

Next LectureLecture 12 · Vertical and Horizontal Federated Learning

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

Prerequisite Knowledge

Previously Covered in This Subject

11.1 Combining Client Models When Data Is Not Shared

11.1.1 Why Simple Mixing Is Not Enough

11.1.2 Non-IID Data and Client Drift

11.1.3 Ordinary Federated Averaging

11.1.4 Student Questions and Answers

11.1.5 Industry Applications and Exam Notes

11.2 SCAFFOLD: Stochastic Controlled Averaging

11.2.1 What “Controlled” Adds on Top of Averaging

11.2.2 Symbols Used Throughout SCAFFOLD

11.2.3 Student Questions and Answers

11.2.4 Exam Notes

11.3 Gradients as Directions: A Three-Weight Toy Model

11.3.1 Loss Sensitivity for Three Weights

11.3.2 Mini-Batch Gradients on Client 1

11.3.3 Other Clients and the Server Average Direction

11.3.4 Student Questions and Answers

11.3.5 Exam Notes

11.4 Control Variates and the SCAFFOLD Correction

11.4.1 Current Gradient Versus Usual Gradient

11.4.2 Remove Local Drift, Add the Global Average

11.4.3 Why Subtract ci​, Not Only Add c

11.4.4 Student Questions and Answers

11.4.5 Exam Notes

11.5 A Full SCAFFOLD Round With Two Clients

11.5.1 What the Server Sends

11.5.2 Client 1: Corrected Steps and New Control

11.5.3 Client 2: The Same Recipe With Different Habit

11.5.4 Student Questions and Answers

11.6 Server Aggregation and Partial Participation

11.6.1 Averaging the Model Changes

11.6.2 Averaging the Control Changes, Then Scaling by Who Showed Up

11.6.3 Intuition of the Whole Loop

11.6.4 Student Questions and Answers

11.6.5 Exam Notes

11.7 FedProx: A Rubber-Band Penalty on Distance

11.7.1 Rubber-Band Picture

11.7.2 Penalty Gradient From a Quadratic Term

11.7.3 Numeric Walkthrough With μ=2 and η=0.1

11.7.4 Local Objective and the Role of μ

11.7.5 Student Questions and Answers

11.7.6 Industry Applications and Exam Notes

11.8 SCAFFOLD Versus FedProx

11.8.1 Control Strength

11.8.2 Extra Vectors Versus On-the-Fly Distance

11.8.3 Gradient Direction Versus Distance

11.8.4 Objective Change Versus Gradient Rewrite

11.8.5 Student Questions and Answers

11.8.6 Exam Notes

Exam Guidance Summary

Key Industry Applications

DML Lecture 11 notes · SCAFFOLD and FedProx: Controlling Client Drift

Sections Breakdown

Exam Revision Notes

Combining Client Models When Data Is Not Shared

SCAFFOLD: Stochastic Controlled Averaging

Gradients as Directions: A Three-Weight Toy Model

Control Variates and the SCAFFOLD Correction

A Full SCAFFOLD Round With Two Clients

Server Aggregation and Partial Participation

FedProx: A Rubber-Band Penalty on Distance

SCAFFOLD Versus FedProx

Was this lecture useful?

BitsNotes AI Assistant

Configure AI Chat

Sign in to use AI Chat

11.4.3 Why Subtract $c_{i}$ , Not Only Add $c$

11.7.3 Numeric Walkthrough With $μ = 2$ and $η = 0.1$

11.7.4 Local Objective and the Role of $μ$