FedBSS Curriculum, Uncertainty Split, and FedCVT

This lecture builds a client-side curriculum for federated photo models, then sketches a second method that fills missing feature views. The first method, FedBSS (federated learning with bias-aware sample selection), scores each private photo, splits easy from hard at a peak uncertainty, and admits hard photos on a cosine schedule so local copies do not yank the global average. The second method, FedCVT (federated cross-view training), uses attention over aligned people, three-classifier pseudo labels, and a joint loss loop. Data stay on the device throughout.

16.1 Federated Setup and Euclidean Model Drift

16.1.1 What federated learning keeps on the device

Federated learning is a training setup that never ships raw examples to a central store. Data never leave the device. A server is the central machine. It holds a global model — one shared set of weights that all clients start from. The server sends copies of those weights to clients. Clients may be many. The working picture here is mobile phones.

Each phone trains on its own photos and labels. That local mix is non-IID — not independent and identically distributed across phones. One phone may hold mostly calm, well-framed shots. Another may hold harsh, unusual shots. Because each phone updates from its own mix, its copy of the weights can move away from the copy it received. The server never sees the pixels. It only later sees updated weights.

This is the same federated rule used in earlier algorithms: send a copy, train locally, send weights back. In FedAvg (federated averaging), the server then takes a size-weighted mean of the returned copies. If client $k$ trained on $n_k$ examples and the selected clients together hold $n={\sum }_k{n}_k$ examples, the next global weights are

$$w^{t+1}=\sum _k\frac{n_k}{n}{w}_k^t.$$

Here $w^{t+1}$ is the new global weight vector, $w_k^t$ is client $k$'s trained copy after round $t$, and the fraction $n_k/n$ is that client's share of the pooled sample count. When every phone holds the same number of photos, this reduces to a plain mean. What changes in this unit is how far a local copy is allowed to walk before that mean is taken, and how we measure that walk.

A local album is like one kitchen's pantry. A calm indoor-cat album is mild spice. A night-time motion-blur album is a heavy hand. The global model is the mailed card. FedBSS will later gate which photos may tweak the spices on a given local epoch. First we need a number for how far the spices moved.

16.1.2 Drift as a distance between two weight sets

A drift is how far a local model moves away from the global model it started as. Drift is a distance. It is the difference between two sets of weights.

Call the received global copy $V$. Architecture stays the same on every client, so layer shapes match. After local training the client holds a new tensor set $W$. Layer 1 has a matrix of weights. Layer 2 has another. Layer 3 has another. For each matching entry we take the difference, square it, and add. The spoken procedure is: take the difference between this copy and that copy, square those gaps, keep adding, and fire them into one number. That number is the drift.

Write the sum of squares as one scalar:

$$D_{\text{sq}}(W,V)=\sum _{\ell }\sum _{i,j}(W_{ij}^{(\ell )}-V_{ij}^{(\ell )}{)}^2.$$

Here $\ell$ indexes layers, $i,j$ index entries in that layer, $W^{(\ell )}$ is the trained local weight tensor, and $V^{(\ell )}$ is the matching tensor from the received global copy. Both live in the same shape as the layer. The result is one scalar, not a matrix.

The spoken name for this number is Euclidean distance. True Euclidean distance also takes a square root:

$$D_2(W,V)=\sqrt{D_{\text{sq}}(W,V)}.$$

The classroom walk squares and adds until one number remains. It does not say the square root out loud. For ranking "how yanked is this update," $D_{\text{sq}}$ and $D_2$ order clients the same way because the square root is strictly increasing for nonnegative inputs. Exam answers should name Euclidean distance across all layers and show the square-and-add procedure that was taught. If a prompt asks for the geometric length in weight space, include the square root.

Notation note: papers often write client drift as the gap between a local update and the global update, $\Delta w_k-\Delta w$. That is the same geometric idea. The exam form in this lecture is the distance between the received global weights and the trained local weights.

The ultimate aim is to reduce the drift. Too much drift makes the later average worse. The spoken warning is that too much of it makes the average result fail: the merged model stops working well.

Drift is not only how far. It is also the direction. Each phone's data pulls its weights along a different ray in weight space. One client may walk one way. Another may walk the opposite way. The global copy is one point. Local copies become other points.

Picture a 2-D slice of weight space. The horizontal axis is weight 1. The vertical axis is weight 2. The global copy sits at a marked origin for this round. After local training, each phone is a point some distance away. The length of the arrow is $D_2$. The heading of the arrow is the direction of the pull. FedBSS will later try to shorten those arrows by delaying the photos that yank hardest.

16.1.3 Worked examples: gentle update and yanked update

Both phones start from the same scalar picture of a weight, used as a teaching stand-in for a full network.

Setup (shared start). Global model weight received: $3.0$. Same architecture on every phone. Local step uses ordinary SGD,

$$w\leftarrow w-\eta \nabla L,$$

where $\eta$ (eta) is the learning rate (step size) and $\nabla L$ is the gradient of the sample loss with respect to that weight.

The same start can produce a very different landing if the photo is a shock. Keep $\eta =0.1$ so only the gradient, and with it the loss, changes.

A yank is the informal name for a large weight update caused by a high-loss sample. High loss $\to$ large gradient $\to$ large SGD step $\to$ large drift.

A two-weight toy makes the tensor sum less abstract. Let the received copy be $V=(3.0,3.0)$. After a yanked step let $W=(0.8,1.1)$. Then

$$\begin{array}{rl}{D}_{\text{sq}}(W,V)& =(0.8-3.0{)}^2+(1.1-3.0{)}^2\\ & =(-2.2{)}^2+(-1.9{)}^2\\ & =4.84+3.61\\ & =8.45,\\ D_2(W,V)& =\sqrt{8.45}\approx \mathrm{2.91.}\end{array}$$

The scalar classroom pair $3.0\to 0.8$ is the one-dimensional slice of this same geometry.

16.1.4 Direction, opposing pulls, and a broken average

Imagine several phones around the same start point. After local training, their copies sit on different rays. Two strong pulls can point against each other. They partly cancel. The average then lands somewhere that suits neither phone well.

The harm is not one phone moving. The harm is many phones drifting at once in conflict directions. Mutual pulling apart is why drift hurts the global model. The server still averages. The average is a compromise among fights, not a model that fits anyone's data.

A numeric cartoon with two clients and one weight makes the cancel visible. Start at $3.0$. Client A yanks to $0.8$ (step $-2.2$). Client B yanks the other way to $5.2$ (step $+2.2$). The plain average is

$$\frac{0.8+5.2}{2}=\mathrm{3.0.}$$

The server thinks nothing happened. Each client just spent compute on a large local move that the other client undid. Neither client's violent seven is now encoded in the global copy. If both had taken the gentle $3.05$ path, the average would have been $3.05$ and both would have kept a small, shared improvement.

On a plot of "weight after local training" (horizontal) versus "round index" (vertical), opposing arrows of equal length meet back at the start. The takeaway is: length without agreement wastes the round.

On-device photo models on phones see this every round. One user shoots calm indoor cats. Another shoots night-time motion blur. Both start from the same download. Both upload very different weights. A plain average can satisfy neither user.

16.1.5 Industry applications

Cross-device federated training on mobile phones is the running setting. Clients are $N$ devices. Each device keeps private photos. The server only merges weight copies. Keyboard, camera, and sensor streams on a phone also change over calendar time, so the same user is not a fixed dataset. Gboard-style next-word models and on-device photo classifiers are the named pattern: train where the pixels live, average only the weights.

FedBSS will sit on the client and change order and timing of local photos. It does not replace the server mean. That split of labor is why a production FedAvg stack can keep its aggregator and still adopt the curriculum later in this lecture.

16.1.6 Exam notes

16.2 Softmax Loss, Spread, and Uncertainty Scores

16.2.1 Forward pass and the true-class probability

FedBSS does not train the received copy on a raw shuffle of local photos. First it scores every local photo with that copy. Scoring is a read-only look: the weights stay frozen while each image goes through the net.

Procedure:

1. Take the global copy just received.
2. Run a forward pass on each local image. No local update yet.
3. Read the softmax vector — class scores that add to one.
4. From that vector, read three working numbers per photo: probability of the true class, loss, and an uncertainty score built from spread.

Softmax turns the last linear scores into a probability vector $p=(p_1,\dots ,p_C)$ with $p_c\ge 0$ and ${\sum }_c{p}_c=1$. Here $C$ is the number of classes. The demo uses three classes: cat, dog, bird. If the last layer emits raw scores $z=(z_1,\dots ,z_C)$, then

$$p_c=\frac{e^{z_c}}{{\sum }_{k=1}^C{e}^{z_k}}.$$

Each $z_c$ is a logit (raw class score before the probability squash). The exponential is always positive, so every $p_c$ is positive, and dividing by the sum forces the vector to add to one. Softmax is a bag of class marbles whose counts add to $100\%$.

$p_{\text{true}}$ — also spoken as $p_2$ / "p true" — is the probability the model gave to the correct class. If the photo is a cat and softmax puts $0.92$ on cat, then $p_{\text{true}}=0.92$.

16.2.2 Loss as minus the natural log

Loss here is a single number for how wrong the model is on that sample. The verbal form is: loss is minus natural log of $p_{\text{true}}$, and loss equal to minus of natural logarithm.

$$L=-\ln p_{\text{true}}$$

where $p_{\text{true}}\in (0,1]$ is the softmax mass on the true class, and $L\in [0,\infty )$. If the mapping is right and confident, $p_{\text{true}}$ is near $1$ and $L$ is near $0$. Natural log is the base-$e$ log. The spoken name is natural logarithm, not log base 10.

Why minus log? If $p_{\text{true}}=1$, then $\ln 1=0$, so $L=0$. If $p_{\text{true}}$ shrinks, $\ln p_{\text{true}}$ goes negative, and the minus sign turns that into a positive penalty. As $p_{\text{true}}\to 0^{+}$, $L\to +\infty$. A model that puts almost no mass on the true class is charged an unbounded fine.

Standard one-hot cross-entropy on a single example is $L=-{\sum }_c{y}_c\ln p_c$. With a one-hot $y$, every term but the true class is zero, so this is the same as $L=-\ln p_{\text{true}}$. Some texts write $\log$ for $\ln$. This lecture uses natural logarithm.

Limiting check: $p_{\text{true}}=1\Rightarrow L=0$. Domain check: $L\ge 0$ whenever $0<p_{\text{true}}\le 1$.

16.2.3 Spread and uncertainty

Spread is the gap between the largest and smallest softmax values on that photo:

$$s=p_{\max }-p_{\min }$$

where $p_{\max }={\max }_c{p}_c$ and $p_{\min }={\min }_c{p}_c$, both in $[0,1]$, so $s\in [0,1]$. The smallest softmax mass is part of the definition, not only the largest.

Uncertainty is one minus spread:

$$u=1-s=1-(p_{\max }-p_{\min }).$$

The verbal walk is: spread is the difference between the maximum value and the minimum value, $92$ minus $3$, which is $89$; uncertainty is one minus spread, so $1$ minus $0.89$ is $0.11$.

When softmax is peaked on one class, spread is large and uncertainty is small. When softmax is flat, spread is small and uncertainty is large. A flat softmax is the confused region.

Notation note: the FedBSS paper writes the same uncertainty as $\alpha (x_i;\vartheta )=1-(\max p-\min p)$. This lecture already reserved $\alpha$ for the cosine mixer in the next sections, so we keep $u$ for uncertainty and $\alpha$ for admission fraction. They are different numbers.

On a sketch whose horizontal axis is "position in a loss-sorted album" and whose vertical axis is $u\in [0,1]$, the curve often rises toward a hump and then falls. That hump is the peak used in §16.3. Landmarks: left tail near $u\approx 0$ (peaked and correct), crest near $u\approx 1$ (flat), right tail maybe low $u$ again (peaked but on the wrong class).

16.2.4 Worked examples: cat photo and confused bird photo

Walk every arithmetic line. Keep percents and probabilities consistent: $92\%=0.92$.

A second photo shows the other end of the scoring ruler.

One photo is not a curriculum. After every local image has a loss, the demo lines the album up as a single increasing list.

16.2.5 Why loss predicts drift

The chain is short and causal:

1. A photo goes through the current weights.
2. Loss $L=-\ln p_{\text{true}}$ is large when the copy is wrong or unsure on that photo.
3. The gradient of $L$ w.r.t. the weights grows with that error.
4. SGD subtracts a step $\eta \nabla L$. A large gradient is a large step.
5. New weights minus received weights is the drift.

For the scalar cat versus yanked-seven picture of §16.1, the same $\eta =0.1$ turned $\nabla L=0.5$ into drift $0.05$ and $\nabla L=22$ into drift $2.2$. Loss is the number that grew that gradient.

Verbal description: loss impacting the gradient, gradient impacting the weight update, and weight update results into new weight, and the difference between these two is the drift. Loss tells you the drift. If loss is high, loss is proportionate to drift.

So FedBSS never needs to form the full Euclidean tensor distance on every photo in order to rank photos. It ranks by loss, which is cheaper and tracks the yank that photo would cause.

Keyboard and camera models on phones already run a forward pass to show a prediction. FedBSS reuses that same pass, before any local SGD, as a hardness meter for the private album.

16.2.6 Exam notes

16.3 Loss Ranking and the Peak-Uncertainty Cut

16.3.1 Sort by loss and rank samples

After every local photo has a loss, line the photos up gentlest to hardest by loss. Gentlest means lower loss. Hardest means higher loss. The paper writes a sample id $X_i$ and a rank id $S_j$. $X_4$ is sample number 4. $S_3$ means that sample is the third-lowest loss after sorting. Position $S_1,S_2,S_3,\dots$ is the rank. Sample under rank is the photo that occupies that slot.

Write the sorted loss list as

$$S_1\le S_2\le \cdots \le S_m,$$

where $m$ is the number of local photos and $S_j$ is the loss of the photo in rank $j$. Ties can be broken by sample id. The six-photo demo of §16.2 already produced

$$S=(0.08,0.16,0.36,1.05,1.30,3.00).$$

Simple photos (low loss, small later update) are marked green in the demo. Photos that yank the weights are marked red. Green means the sample does not move the network much. Red means the sample yanks the network. The algorithm will not feed the received copy the raw bag of photos. It will make two copies of the idea of the set: an easy pool and a hard pool. Then it will train easy to farthest.

16.3.2 Split at the peak uncertainty, not a fixed threshold

Walk along the loss-sorted list and read each photo's uncertainty $u=1-(p_{\max }-p_{\min })$. Uncertainty often rises, hits a high point, then falls. That high point is the peak. The demo's peak was $0.95$. The number $0.95$ is not a magic global cutoff. The peak is what matters. You may pick any numeric peak that the client's own list actually shows.

Let $j^{\star }$ be the rank where $u$ is largest:

$$j^{\star }=\arg \underset{j}{\max }u(S_j).$$

Photos with rank $j\le j^{\star }$ sit on the lower-loss side of the crest. Photos with rank $j>j^{\star }$ sit on the higher-loss side. Above the peak (lower-loss side): green, unbiased, simple. Below the peak (higher-loss side): red, biased, hard. "Unbiased" here means photos the current global copy already handles. "Biased" means photos that drag the copy toward this phone's odd mix.

The FedBSS paper writes the same rule as $\alpha (S_{\text{mid}}^n)={\max }_i\alpha (x_i^n;\vartheta )$, where that paper's $\alpha$ is our $u$. The cut is the sample of highest uncertainty, not a shared numeric threshold.

A toy walk for six photos makes the crest visible. Suppose the loss-sorted uncertainties are

$$u=(0.11,0.28,0.60,0.95,0.72,0.40).$$

The maximum is $0.95$ at rank $4$. Green pool: ranks $1$--$4$. Red pool: ranks $5$--$6$. If another phone's list peaked at $0.70$ on rank $2$, that phone would cut in a different place. Same algorithm, different cut. That is the point of a peak.

On a sketch, the horizontal axis is rank $S_1$ to $S_m$ (easy to hard). The vertical axis is uncertainty $u$. The shape is a hump: low, rising, crest, falling. The landmark is the crest. Split there. Takeaway: the cut travels with the client's own hump, not with a posted number.

16.3.3 Confident-correct, confused, then confident-wrong

Walk left to right on the loss-sorted list.

1. Confident and correct. Softmax is peaked on the true class. Example: $0.92$ on cat when the photo is a cat. Loss is small. Uncertainty is small. These are green.
2. Confused, peak uncertainty. Softmax becomes flat. The model is not sure. Uncertainty hits its maximum. This is the yellow cut. Split here.
3. Confident and wrong. Softmax may peak again, but on the wrong class. Loss is large because $p_{\text{true}}$ is small. The model can look sure while being wrong. These are red.

The transition from correct to wrong passes through the peak of uncertainty. That is why the cut sits on the peak, not on a loss number alone. High-loss red photos drag the model toward this phone's mix. Low-loss green photos are ones the model already handles. They will not cause much drift.

A numeric cartoon of zone 3: suppose bird is the true class and softmax is $(0.03,0.92,0.05)$ on (bird, cat, dog). Then $p_{\text{true}}=0.03$, $L=-\ln (0.03)\approx 3.51$, $s=0.92-0.03=0.89$, $u=0.11$. Uncertainty is low again, but loss is huge. A threshold on $u$ alone would miss this photo. A peak-after-sort still places it on the hard side because loss already lined it up on the right.

16.3.4 Paper ranking from sample id to sort id

A worked order from the paper diagram, eight samples:

Visual: imagine eight photo thumbnails in capture order $X_1,\dots ,X_8$, then a second row that permutes them into $S_1,\dots ,S_8$. The permutation is what the sort computes. Color the left block green and the right block red at the uncertainty crest.

16.3.5 Student questions and answers

Several students asked why the cut is a peak rather than a posted number. Collapse those repeats into one answer.

The three-zone walk is a second, distinct confusion: people mix up "unsure" with "wrong."

16.3.6 Exam notes

16.4 Cosine Alpha Schedule and Easy-to-Hard Admission

16.4.1 Easy-to-hard mixing with alpha

The algorithm says: train easy to hard on a cosine schedule. Green photos are always in. They are ready from epoch 1. Red photos enter gradually over local epochs. Alpha is the mixing fraction that decides how much of the red set is admitted this epoch. Alpha varies from $0$ to $1$.

Think of school years. LKG and UKG first, then first year, then later years. You slowly raise complexity. You do not hand a child the hardest book on day one. Green photos are LKG: always in the room. Red photos are later grades: they enter when the class is ready. The analogy breaks if you treat a red photo as "failed" and send it home. Every photo still graduates by the last epoch.

Green photos stay in the training set every epoch. Alpha only gates the red photos. At the end of the local epoch budget, every photo is in. Nothing is thrown away. The trick is timing. By the time the hardest photo enters, the copy has already trained on simple photos and on milder reds. Its drift on the hardest shot is then smaller. If you dump the hardest shot first, drift jumps.

16.4.2 Cosine schedule formula

Verbal description: "what is the formula one minus cos of phi e by e by 2 slow start" and later "alpha c equal to a phi into e by e by 2" with "e is the current epoch and E is the total number of epochs." The intended shape is slow start, faster in the middle, gentle finish. Spoken "phi" is the circle constant $\pi$, not the golden ratio.

The reconstruction that matches the spoken table (epoch 1 $\approx 0.02$, epoch 3 $\approx 0.21$, epoch 4 $=34.55\%$, later $\approx 65\%$, last epoch $=100\%$) is

$$\alpha (e)=\frac{1-\cos (\pi e/E)}{2}$$

where $e\in \{1,\dots ,E\}$ is the current local epoch, $E$ is the total number of local epochs, $\alpha (e)\in [0,1]$, and $\pi$ is the usual circle constant.

Why this shape? At small $e$, $\pi e/E$ is a small angle, $\cos \approx 1$, so $1-\cos$ is near $0$ and $\alpha$ crawls. In the middle, cosine falls through zero and $\alpha$ steepens. At $e=E$, $\cos (\pi )=-1$, so

$$\alpha (E)=\frac{1-(-1)}{2}=1.$$

All red photos are eligible. A linear gate $e/E$ would admit reds at a constant clip. The cosine gate spends extra early epochs on greens, which is what the album's dense-near-the-cut shape needs.

The FedBSS paper writes the same mixer as ${\mathbf{X}}_{t,e}^n={\mathbf{X}}_{\text{no}}^n+\alpha {\mathbf{X}}_{\text{bias}}^n$ with that cosine $\alpha$. Unbiased (green) plus a cosine slice of biased (red). Standard form matches the classroom formula.

16.4.3 Worked epoch table and how many red samples enter

Take $E=10$ local epochs, as in the demo ("let E equal to 10").

How does a fraction become a count? One spoken rule is: round the product of alpha and a sample count. At epoch 4, $34.55\%$ of six photos is $0.3455\times 6=2.07$, which rounds to 2. A later spoken walk for six photos:

• Epoch 0: select none.
• Early epochs: about 10–20%, still about one extra slot.
• Epoch 4: $34.55\%$ $\to$ about two.
• Epoch 5: three.
• Epoch 6: four.
• Epoch 7: five.
• Epochs 8–10: all six.

That walk multiplies alpha by all six photos. A correction from class: green photos are always in, so alpha should multiply only the red count, not the full six. If three photos are red, use $\mathrm{round}(3\cdot \alpha (e))$, not $\mathrm{round}(6\cdot \alpha (e))$. Using six makes the first red look like it enters too early (around epoch 3). Using the red count delays the first hard photo (spoken as the fourth overall sample, $S_4$) toward later epochs, including a telling that tough $S_4$–$S_6$ start near epoch 6.

Write the intended gate as

$$n_{\text{red}}(e)=\mathrm{round}(|R|\cdot \alpha (e))$$

where $|R|$ is the number of red photos after the uncertainty cut, and $n_{\text{red}}(e)$ is how many red photos may join the greens in epoch $e$.

For $|R|=3$:

A simplified milestone picture used while teaching: introduce more hard photos around epoch 4, then around epoch 7, then by epoch 10 alpha is 1 and every sample is in.

You must pick $E$ large enough that alpha can reach 1 while still leaving room to bring every tough photo in. If you only run three local epochs, you may train only the easy four and never admit the rest. Covering the whole set is a design choice on $E$, not a separate fancy optimizer. Trial runs can set $E$. One spoken lower bound in that six-photo picture was: you may need on the order of eight epochs before every tough sample has had a chance to enter.

On a plot of epoch $e$ (horizontal) versus $\alpha (e)$ (vertical, $0$ to $1$), the curve is an S: flat, then steep, then flat. Landmarks: $\alpha (5)=0.5$ at the midpoint, $\alpha (10)=1$ at the end. Takeaway: slow start, not a linear ramp.

16.4.4 Green always in; do not throw samples away

Green photos are always in. Red photos enter gradually, controlled by alpha. You are not discarding hard photos. You are delaying them. Every image is used by the end of the local run.

Order inside an epoch is easy to hard: train the current mix from gentlest remaining toward the newly admitted reds. After local training, the phone uploads its weights. The server does not change its merge rule. It still takes a plain average and starts the next communication round.

Over rounds, the red set can shrink by itself. Hard photos become easier once the model has seen milder versions. Next round, those photos may no longer yank as hard. Classification into red versus green can change. Drift on former reds gets smaller. That is the "global progressive learning" picture in FedBSS: the biased set shrinks as the global copy absorbs those patterns.

Time cost: each epoch still runs SGD on the current mix. Early epochs are cheaper (fewer reds). The last epochs cost about the same as ordinary local training on the full album. Space cost is one extra pass to score and sort, then two index lists $G$ and $R$.

16.4.5 Yank, timing, and why later epochs hurt less

The bar next to each photo in the demo is the loss that photo induces, and also its yank. Yank is the informal name for a large weight update caused by a high-loss sample. High loss $\to$ large gradient $\to$ large SGD step $\to$ large drift.

Same photo, two timings:

• Epoch 1 dump. If the fourth-hardest photo is trained immediately, the copy has not yet learned the simple three. Drift is a big yank.
• Late admission. If that same photo enters after epochs spent on $S_1,S_2,S_3$ (and maybe milder reds), the copy has already seen nearby patterns. Loss is smaller. Drift is smaller. By an epoch-9 picture of a later hard shot, the model has already seen four and five, so seeing six does not yank as much.

Directly introducing $S_6$ first causes more drift. Incremental introduction causes less. That is the whole point of the cosine gate.

A generated numeric demo may print a slightly wrong formula or multiply by the wrong count. The idea to keep is: greens always in, reds admitted by a slow-then-fast-then-gentle $\alpha (e)$, round to an integer count, never drop photos, upload, average.

16.4.6 Student questions and answers

The epoch-budget question is the first distinct confusion: greens-first training can hide the hard tail if $E$ is tiny.

Alpha itself is a second confusion: people hear "percentage" and do not see the cosine.

Turning a table of ten alphas into a calendar of admissions is a third confusion.

The demo's multiply-by-six table is the correction that students tripped on.

The last distinct question is the same photo at two clocks.

16.4.7 Exam notes

16.5 Warm-up Rounds and Plain Server Averaging

16.5.1 Why a fresh model cannot rank samples

FedBSS is not applied on a brand-new random copy. Stage one is a warm-up. Stage two is the curriculum (sort, peak split, cosine mix).

Warm-up length used here: 50 communication rounds. Spoken variants were "50 epos", "50 uploads", "50 samples", and "50 pokes". They all point to about 50 server rounds of ordinary local training and upload, without the curriculum gate. During warm-up you do not run the second algorithm. You do not sort. You do not split.

Why wait? A fresh model has near-random weights. Losses are meaningless noise. If you sort or split now, the split jumps around widely every round. Every sample may look similarly hard, or the ranking may shuffle at random. You would mis-label simple as hard and hard as simple. The ranking would be meaningless and unstable.

Warm-up trains until losses carry real difficulty, not noise. Weights move off the random start. Then curriculum learning can read loss and uncertainty as genuine hardness.

The FedBSS paper names this the diversified knowledge acquisition stage: ordinary aggregation, used as a cold start so the later uncertainty peak does not thrash. Their experiments set warm-up rounds to $50$ out of $200$ total rounds. $50$ is the working number in this lecture. It is not a theorem that $50$ is always best. Zero warm-up is worse because the peak jumps. A huge warm-up is also worse because the second stage then has little room left.

SCAFFOLD and FedProx (spoken as "fed box") as studied earlier did not look into this sample-level drift split. They regularize or correct client drift in other ways. FedBSS adds the warmup-then-curriculum attack on which photo yanks the copy.

When to pick which: use FedBSS when you can score and sort the local album and you want a sample-level curriculum. Use FedProx or SCAFFOLD when you want a regularizer or a gradient correction and you will still shuffle the album as usual. They can sit beside each other. They are not the same knob.

16.5.2 Two-stage client procedure

Warm-up trains until weights stabilize. Curriculum then reads those stable losses. Every client follows that order.

Those are the two stages: warm-up first, curriculum second. There is no extra stopping rule taught here such as "stop warm-up when loss shows learning difficulty." The working instruction is: run the 50-round warm-up, then switch. Do not hunt a fancy loss threshold that was not derived in this session.

After local FedBSS training, each phone uploads its local model. The server does ordinary aggregation. No new server trick. It takes a plain average of the uploaded weights. That average is the next global copy. Next round starts.

16.5.3 Repeat warm-up when local data grows

After the first global average, the next training session still uses the same two-stage idea. Local data is not frozen. New sensor readings arrive. New cat photos and new sevens appear on the phone. That fresh mass can yank a copy that was stable on last week's album.

So: let the new data be seen during a warm-up so weights adjust, then run curriculum again. You could try jumping straight to curriculum on round two. It is not a crime. Warm-up is still better because the new pile can recreate the same ranking-noise problem.

A phone that collected a week of night-time clips is not the same client that warmed up on indoor cats. The global copy may be decent on last week's mix and still be random-like on the new clips. Scoring the new clips immediately would paint almost all of them red, or shuffle the peak. A short ordinary-training warm-up lets the copy see the new pile before the cosine gate starts delaying it.

16.5.4 Server aggregation stays a plain average

Score each photo's loss. Sort. Cut on uncertainty. Train greens then admitted reds. Then average. That average is the new global model. Drift on the client is still the Euclidean-style distance between the received global copy and the trained local copy, summed across all weights.

If $K$ phones upload copies $w_1,\dots ,w_K$ with sample counts $n_1,\dots ,n_K$, FedAvg still sets

$$w\leftarrow \sum _{k=1}^K\frac{n_k}{n}{w}_k,n=\sum _{k=1}^K{n}_k.$$

FedBSS changes local order and timing. It does not replace the server mean. A production aggregator can keep this line of code.

16.5.5 Student questions and answers

The first question is whether warm-up is global or per client.

A second, distinct hope is an automatic halt.

Non-stationary data is the third confusion.

Some students treated the 50 rounds as the curriculum itself.

The last question is skipping warm-up once a global copy exists.

16.5.6 Industry applications

Phones keep collecting camera rolls and sensor streams between communication rounds. A keyboard or vision model that warmed up last week still sees a new private pile this week. Repeating a short ordinary-training warm-up before the easy-to-hard gate matches that non-stationary on-device data.

FedAvg-style servers can keep their plain mean. FedBSS sits on the client. SCAFFOLD and FedProx remain available as other drift tools, but they are not this sample-sort curriculum.

Cross-device photo classifiers (cats, dogs, birds, handwritten sevens) are the running named setting. The same warm-up-then-curriculum pattern fits any on-device stream whose private mix shifts by the week.

16.5.7 Exam notes

16.6 FedCVT Attention, Pseudo Labels, and Cross-View Training

16.6.1 Missing raw features and missing representations

The last method is FedCVT (federated cross-view training; the paper is also referred to as CVT). It is exam-relevant. The pass here is high-level, not a full derivation.

Setting: several parties or people, each with raw features. For some people, some features are missing. A neural net maps raw features to representations — higher-level vectors. If the raw features are missing, the representation is also missing. Example: persons 1, 2, 3, 4. Person 4 has holes. There is still a label column for people who are aligned.

Aligned people have overlapping observed features. Non-aligned people have missing blocks. Party B may be missing one block of coordinates. The job is to fill missing views and still train a classifier.

This is vertical federated learning: parties hold different feature blocks of the same people, and they will not ship those blocks. Horizontal federated learning (FedBSS's phone setting) holds different people with the same feature type. FedCVT is the vertical, missing-view cousin.

A bank that holds account features and a retailer that holds purchase tags is the named picture. They share some customers (aligned) and many unique customers (non-aligned). FedCVT tries to use the unique customers too.

16.6.2 Attention with query, key, and value

To fill a missing representation, use attention. Attention asks: how close is this incomplete person to each complete person?

For person $P_4$, learn a query vector $q_4$. Aligned people supply keys $k_j$ and values $v_j$. Compare the $P_4$ query to each aligned key. Apply softmax over those match scores. Mix the values with those softmax weights.

Verbal description: take this $P_4$ query to this key, apply the softmax technique, and compute the values. A numeric cartoon used in the room: $P_4$ is about $60\%$ of $P_1$, $30\%$ of $P_2$, and the rest of a third person. One spoken fragment said "$110\%$ of this value," which cannot be a set of softmax weights. Softmax weights are a convex combination: they are nonnegative and add to $1$. A leftover $10\%$ on $P_3$ completes $60+30+10$.

The filled vector is a soft nearest-neighbor in representation space, not a guess in raw pixels alone.

Standard form (from the FedCVT write-up) inserts the usual attention scale $1/\sqrt{d_k}$:

$$g(Q,K,V)=\mathrm{softmax}\left(\frac{Q{K}^{\top }}{\sqrt{d_k}}\right)V.$$

The lecture named $Q$, $K$, $V$ and softmax mixing. It did not speak the $1/\sqrt{d_k}$ factor. Keep the classroom form for the exam. Know that the paper's scaled dot-product is the same mixer with a temperature that depends on the key dimension $d_k$. Scaling prevents large dot products from making softmax look like a one-hot.

16.6.3 Three classifiers and pseudo labels

After filling, assign a pseudo label when you lack a true label.

Train three classifiers on different views of the available data:

1. Trained with only one view (spoken "only blue values") — call this $f^A$.
2. Trained with only the other view (spoken "only red values") — call this $f^B$.
3. Trained with both views — call this $f^{AB}$.

Feed the sample to all three. Suppose they vote cat at $80\%$, $70\%$, and $90\%$. Majority says cat, so the pseudo label is cat. Two extra gates were stated:

• At least the three classifiers agree (all three agree).
• Every classifier's confidence is above a threshold greater than $50\%$.

Only then write the pseudo label. Then you can train as if that sample were labeled.

A reject cartoon: votes cat $80\%$, cat $70\%$, dog $90\%$. They do not all agree, even though two say cat and every confidence is high. Do not write a pseudo label. Another reject: all say cat at $40\%$. They agree, but they fail the $>50\%$ gate.

16.6.4 Cross-view training loop

Cross-view training shares representation across models and improves representation learning. Combine several losses and update. Three classifiers give three losses. Update from those losses. Repeat:

1. Learn representations from whatever raw features exist.
2. Fill missing representations with attention over aligned people.
3. Get labels, including pseudo labels that pass the three-way vote and the $>50\%$ gate.
4. Do cross-view training with the combined losses.
5. Repeat.

Named loss symbols $L_{\text{default}}$ and $L_{\text{theme}}$ were asked about in passing. They were not derived in this high-level pass. Read them in the FedCVT paper rather than inventing expansions here.

The paper's named objective, for notation only, is a sum of three cross-entropies plus similarity and difference regularizers:

$$L_{\text{obj}}=L_{\text{ce}}^{\text{fed}}+L_{\text{ce}}^A+L_{\text{ce}}^B+\text{(regularizers)}.$$

That is the write-up's list. It is not a derivation of $L_{\text{default}}$ or $L_{\text{theme}}$. If an exam asks what those two spoken names are, say they belong to the FedCVT write-up and were not expanded in this pass. The working loop is still: learn, fill, label, train, repeat.

FedBSS is treated as the harder method and the one with more problem-style questions. FedCVT is still on the exam, with more left to self-study (spoken as leaving about 60 to 70 percent). The HTML walkthrough, the presentation, and the paper are the study path for the missing algebra.

16.6.5 Student questions and answers

The named-loss question is one confusion. The exam-coverage question is another.

FedCVT still carries an exam item even when a problem sheet is optional.

16.6.6 Exam notes

Exam Guidance Summary

Several items are reason-and-solve, not one-line definitions. If you understand the mechanism you can answer even when a worksheet is not separately posted. Read the FedBSS paper, the presentation, and the HTML walkthrough. The same holds for FedCVT at a slightly higher self-study share.

High-value questions collected in this session:

1. What is drift? Euclidean-style distance between received global weights and trained local weights, with direction as well as size. Too much drift poisons the average. Keep the pair $3.0\to 3.05$ (drift $0.05$) versus $3.0\to 0.8$ (drift $2.2$).
2. Why the peak, not a threshold? Each client's private mix has its own uncertainty peak. A global cutoff such as $0.85$ cannot be shared. $0.95$ in the demo is an example peak, not a constant to memorize.
3. Three softmax zones. Confident+correct; confused (flat softmax, peak uncertainty); confident+wrong.
4. Compute loss and uncertainty. $L=-\ln p_{\text{true}}$, $s=p_{\max }-p_{\min }$, $u=1-s$. Worked checks: $-\ln (0.92)\approx 0.08$, $-\ln (0.35)\approx 1.05$.
5. Paper ranks. Decode $X_i$ versus $S_j$. Example: $X_7=S_1$ simplest, $X_8=S_8$ hardest.
6. Cosine alpha. $\alpha (e)=\frac{1-\cos (\pi e/E)}{2}$, greens always in, reds admitted by a rounded fraction, no sample dropped. Same hard photo yanks less when admitted late. For $E=10$, recover $0.02$, $0.21$, $34.55\%$, $65\%$, and $1$.
7. Why 50-round warm-up? Random weights make losses noise. Sort-and-split would misclassify simple and hard. Warm-up until losses mean real difficulty. No extra stop rule was derived here.
8. Why warm-up again next round? New on-device photos and sensor data.
9. FedCVT. Attention fill of missing views, three classifiers, pseudo labels with all-agree and $>50\%$, cross-view losses. Expect a question. FedBSS still carries more of the problem load.

Write reasons, not only names. Show the numeric substitutions when a table is asked. FedBSS carries more numeric drills. FedCVT is still on the paper. Do not skip either.

Key Industry Applications

On-device federated photos. Phones train cat/dog/bird (and digit) classifiers without uploading pixels. Non-IID camera rolls yank local copies unless hard shots are delayed. FedBSS is the client-side curriculum for that setting: score the private album, split at the local uncertainty peak, admit reds on a cosine clock.

Streaming sensors. Client data evolves between rounds. Warm-up then curriculum is rerun so new sevens and new cats do not immediately dominate SGD. A keyboard model or a camera model that was stable last week still sees a new private pile this week.

FedAvg-compatible servers. FedBSS keeps the server as a plain average. The curriculum is a client-side order and timing policy. An existing aggregator does not need a new merge rule.

Vertical and missing-view collaboration (FedCVT). Parties hold different feature blocks. Attention over aligned people fills a missing party. Pseudo labels let unlabeled rows join training when three view-specific classifiers agree with confidence above one half. A bank-plus-retailer pair that shares some customers and not others is the named collaboration.

Related federated baselines. SCAFFOLD and FedProx address client drift without this sample-level sort. FedBSS adds warmup-plus-curriculum on the album itself. Pick the sample curriculum when photo order is the problem. Pick a proximal term or a control variate when the walk size or the gradient correction is the problem.