Distributed Machine Learning

Prototype-Based Vertical Federated Learning with Unaligned Data

Published: 2026-09-11

Level: postgraduate

Audience: Postgraduate students in Distributed Machine Learning

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

Vertical partitioning — covered in Lecture 1
Aligned and unaligned samples — covered in Lecture 13
Local and global priors — covered in Lecture 13
Adapter, gating and active-party aggregation — covered in Lecture 13

14.1 Aligned and Unaligned Data in Vertical Partitioning

14.1.1 What Vertical Partitioning Means

A bank knows how you spend. A hospital knows how you heal. A shop knows what you buy. None of them will hand over raw rows, yet together they could answer one question about you. How do we learn from all three views without moving private columns?

A vertical partition (split of columns across owners while rows refer to the same people or items) means different parties hold different features for the same people or items. We write a party index as $m$ , where $m = 1, 2, 3$ in the running example with three clients. Each party $m$ holds its own feature matrix $X^{(m)}$ , where each row is one sample and each column is one feature that only that party can see. If party $m$ has $N^{(m)}$ local rows and $d_{m}$ local columns, then $X^{(m)} \in R^{N^{(m)} \times d_{m}}$ . Row $i$ of $X^{(m)}$ , written $x_{i}^{(m)} \in R^{d_{m}}$ , is what party $m$ knows about sample $i$ .

An embedding width (length of the learned vector that leaves each party) will later be written $d$ . It is shared across parties, for example $d = 20$ . An input width (count of raw columns that enter each party) is written $d_{m}$ and stays different per party, for example 40, 60 and 50.

Think of three friends who each know one part of a story. One knows what was bought, one knows where it was bought, one knows when it was bought. Only when they bring their parts together can they answer who bought what. The joint answer needs all parts, but no one wants to hand over private details. The mapping is direct: each friend is one party $m$ , each partial memory is one row $x_{i}^{(m)}$ , and the full answer is the fused vector $z_{i}$ at the server. Where the story breaks: friends can talk in plain words, while parties may only send learned vectors of fixed width, never raw columns.

Real life runs on this split. A bank holds money features such as loan amount and repayment history, while a hospital holds health features such as test results and visit dates. Both describe the same person, but neither side sees the other side raw data. A shop might hold a third view, such as basket size and return rate. All three views refer to one customer identifier, yet the columns never meet in one disk.

Vertical split, active party and label placement. Let $m \in {1, 2, 3}$ index the passive clients and let $S$ denote the active party, the server that holds labels and trains the final decision model. Passive parties or clients only hold features. Labels $y$ , where $y \in {1, 2, 3}$ for cat, dog and elephant in the teaching example, live only at $S$ . Clients never see $y$ . This split shapes every loss and every message that follows.

The teaching rule in symbols: if $E^{(m)}$ is the local extractor of party $m$ and $C$ is the global classifier at $S$ , then unaligned samples improve $E^{(m)}$ while aligned samples with labels train $C$ . In words: unaligned data trains the local network, aligned data trains the global decision model.

In plain steps, the data flow is: each party keeps $X^{(m)}$ local; each party learns a local extractor $E^{(m)}$ from its large unaligned pool; the small aligned slice is passed through the improved extractors; the resulting vectors travel to $S$ ; $S$ fuses them and trains $C$ with labels. No raw $X^{(m)}$ row ever leaves its owner.

Scope: This vertical design applies when identifiers overlap but columns do not. It fits bank plus hospital plus shop views of one customer. Assumption: identifiers can be matched across parties without leaking features, and at least a small aligned slice exists. When there is no overlap at all, there is nothing to fuse, and the method does not apply. When one party already holds all columns, there is no need for federation.

Picture the layout as a table cut the wrong way for a single owner. Draw a wide table whose rows are customers and whose columns are split into three coloured blocks: left block money columns at the bank, middle block health columns at the hospital, right block retail columns at the shop. Draw horizontal lines for identifiers. A row where ink appears in all three blocks is aligned. A row where ink appears in only one block is unaligned. The takeaway in one line: the cut is by columns, and alignment is about whether a row has ink in every block.

Common traps here: (1) mixing up vertical with horizontal splits — horizontal means same columns on many phones, vertical means different columns on few institutions; (2) thinking clients see labels — they never do, only $S$ stores $y_{i}$ ; (3) thinking more aligned rows are always available — in this lecture the running count is 20, so the design must live with few labeled rows and many unlabeled rows.

Questions of privacy often surface at this point. The answer built into the design is that parties send fixed-width learned vectors, not raw columns, and the server learns from those vectors. That is why output width is fixed in code while input quality stays private.

Recap: vertical means same identifiers, different features per party, with labels only at the active party $S$ . Unaligned rows build local strength; aligned rows build the joint decision. This handoff sets up the next step: how to tag a row as aligned or unaligned in practice.

Bank plus hospital plus shop fusion for credit, care or fraud checks without sharing raw columns is the running domain link. Credit teams want repayment plus health-plus-retail signals; care teams want spend plus clinical signals; fraud teams want all three. The vertical pattern lets each team keep its columns while the joint model still sees the full person.

14.1.2 Aligned Samples and Unaligned Samples

If a customer opens a bank account today but visits the hospital next month, is that customer aligned today? The answer decides which training path the row takes.

An aligned sample (identifier present in every party) is one sample identifier that appears in all parties. We write the aligned set as $A$ , with $∣ A ∣ = N_{A}$ . The notation $∣ A ∣$ means count of members in $A$ . In the session the running number was $N_{A} = 20$ aligned samples. An unaligned sample (identifier missing in at least one party) is an identifier that appears only in one party, or in a subset but not in all. We write the unaligned set of party $m$ as $U^{(m)}$ , with size $N_{U}^{(m)}$ .

Aligned set from common identifiers, unaligned sets per specific party. Let $I^{(m)}$ be the identifier list held by party $m$ . Then the aligned set is the overlap across all parties:

$A = I^{(1)} \cap I^{(2)} \cap I^{(3)}$

where $\cap$ means overlap, and $N_{A} = ∣ A ∣$ . The unaligned pool of party $m$ holds the rest of its local identifiers:

$U^{(m)} = I^{(m)} ∖ A$

where $∖$ means members of $I^{(m)}$ not in $A$ . In words: aligned means common to all parties, unaligned means specific to one party or to a subset that is not yet complete. A sample present in $C_{1}$ and $C_{2}$ but missing in $C_{3}$ is still unaligned. If it later arrives at $C_{3}$ , its status flips to aligned. Status depends on presence across parties, not on content alone.

Take three parties $C_{1}$ , $C_{2}$ , $C_{3}$ . If sample $i = A_{1}$ is present in $C_{1}$ and $C_{2}$ and $C_{3}$ , it belongs to $A$ . If sample $j$ is present only in $C_{1}$ , it belongs to $U^{(1)}$ . If a sample is present in $C_{1}$ and $C_{2}$ but not yet in $C_{3}$ , it is still unaligned. If it later arrives at $C_{3}$ , its status can change to aligned.

Work the counts once. Suppose $C_{1}$ has 500 unaligned cat images, $C_{2}$ has 5000, $C_{3}$ has 2. Then the extractor trained on 5000 images sees far more poses, lights and shapes than the one trained on 2 images. This count gap later explains why the server weights one party more than another. The teaching weights near 80, 18 and 2 percent trace back to gaps like 5000 versus 500 versus 2.

The setup is coded by an engineer or admin who loads data into each party store. That setup step records which identifiers overlap. There is no self-discovery magic. When a new record arrives, the system checks its identifier against the other parties and tags it as aligned when it is present everywhere, else unaligned. The tag can change over time as more parties receive the same identifier. Identifier matching itself is done with privacy-safe joins, so the check learns overlap without pulling raw columns.

Scope: the subset rule above is the full test. A row must clear every party to count as aligned. Assumption: identifiers are stable and matchable across parties. When identifiers are noisy or duplicated, the tag is not trusted until the matching step is fixed.

See it as three overlapping circles, one per party. The small lens in the centre where all three overlap is $A$ with 20 members. Each outer crescent is one $U^{(m)}$ . A dot that sits in two circles but outside the third is still outside the lens. When the third circle grows to cover it, the dot falls inside the lens and flips to aligned. One-line takeaway: alignment is all-or-nothing across parties.

Watch for: (1) calling a two-party overlap aligned in a three-party task — it is still unaligned until the third party holds it; (2) thinking the tag is fixed for life — arrival of the missing copy flips it; (3) thinking clients vote on the tag — the engineered identifier check decides, not a learned guess.

Recap: $A$ is the common overlap with 20 rows here; each $U^{(m)}$ holds the local remainder. Counts like 500, 5000 and 2 already hint at unequal trust later. The next step links this split to labels: who holds the right answer.

14.1.3 Where Labels Live and Why That Matters

A label (the right answer for an aligned sample) is written as $y_{i}$ , where $i \in A$ . Only the active party stores $y_{i}$ . Clients store only $X^{(m)}$ rows, never $y_{i}$ . For the teaching classes cat, dog and elephant, $y_{i} \in {1, 2, 3}$ , often coded as a one-hot list with a single 1 at the true class.

Supervised top, distance-based bottom. The global classifier at $S$ can use a supervised loss because it sees pairs $(z_{i}, y_{i})$ , where $z_{i}$ is the fused vector for aligned sample $i$ . Local extractors cannot use labels because their unaligned samples have no $y$ . They must use distance to prototypes instead. That split — supervised at the top, distance-based at the bottom — runs through every later section. In short: the server learns from right answers, clients learn from nearness to reference points.

Why the split matters can be felt in sizes. Twenty labeled pairs $(z_{i}, y_{i})$ are enough to fit a small classifier $C$ but far too few to fit three deep extractors from scratch. Thousands of unlabeled local rows are enough to shape each $E^{(m)}$ if the loss does not need labels. The design matches each data pool to the loss it can feed.

Scope: this label placement fits tasks where the label owner is also the model owner, such as a bank that holds default labels while the hospital and shop hold features. Assumption: aligned identifiers at $S$ carry correct labels. When labels are noisy, the top loss inherits that noise and the whole loop suffers.

Draw two floors. Ground floor: three separate rooms, each with a large pile of unlabeled local rows and a small workbench $E^{(m)}$ . Top floor: one room at $S$ with a small table of 20 fused rows plus 20 labels and the classifier $C$ . Arrows go up: improved vectors rise from each bench to the top table. Arrows come back down: global prototypes and priors return to each bench. Takeaway: labels never go down, vectors never carry raw columns up.

Do not mix up: (1) aligned rows without labels — by definition aligned rows at $S$ carry $y_{i}$ ; (2) unaligned rows with labels — they carry none locally; (3) training $C$ on unaligned rows — $C$ only reads fused aligned rows.

Exam note: expect a short-answer check on who holds what: features per party, labels only at the server, aligned identifiers shared, unaligned identifiers local. State the pair $(z_{i}, y_{i})$ for the top loss and the label-free distance loss for the bottom, and the mark is safe.

Hospitals that hold outcome labels while banks hold spend features flip the same pattern: whoever holds $y$ becomes $S$ . The math stays the same, only the owner of the top floor changes.

14.1.4 Student Questions and Answers

Q: Different parties have different features, like a bank with money data and a hospital with health data. Their vectors will look different. How does the active party join them for aligned training?

A: The vector spaces start out different because the input qualities differ, money versus health. The code fixes the output size so all parties emit the same width, for example 20 numbers per vector, so shapes match and representations can be joined at the active party. What cannot be fixed by code is the spread or distribution of values. Those different spreads of money and health vectors are mapped to a shared space by a small network at the server, called the adapter, and then mixed with learned weights in gating. The server learns how to place money-based vectors and health-based vectors into one shared area where one fused vector can stand for one aligned sample.

The question is natural because money columns and health columns live in different units. The fix has two halves that students should keep apart: width is fixed by design, spread is learned by the adapter. Width matching lets vectors stack; adapter mapping lets them mix with sense. Gating then decides trust per sample.

Q: How does a client know that a given sample is unaligned? Only the server sees all parties.

A: No client decides this alone. During setup, the team that loads the data collects identifiers from all parties and marks which identifiers are present everywhere. That mark is stored with the data. When a new record comes in, the system checks whether its identifier is present in the other parties. If it is present in all three, it is tagged aligned. If it is missing in one or more, it stays unaligned. If it arrives later at the missing party, the tag can flip to aligned. It is an engineered check on identifiers, not a learned guess. Setup loads data, marks identifiers, and flips the tag when coverage becomes complete.

A frequent follow-up is whether clients learn the tag by training. They do not. Tagging is a data step before training, done on identifiers alone. Training then routes the row: unaligned rows stay local for extractor shaping, aligned rows travel as vectors for joint learning.

Recap: vertical splits keep raw columns local; aligned means present in all parties with labels at $S$ ; unaligned means the rest and trains extractors without labels. The bridge to the next section: with labels out of reach locally, what signal trains each $E^{(m)}$ on its large unaligned pool.

14.2 Local Feature Extractors Trained on Unaligned Data

14.2.1 Extractor Network and Representations

Twenty labeled rows cannot train three deep networks. Thousands of unlabeled local rows can — if we give each party a loss that needs no labels. What should that loss look at?

A feature extractor (small neural network inside each party that turns raw local columns into a compact vector) is written as $E^{(m)}$ , with parameters $θ^{(m)}$ . For a local row $x_{i}^{(m)}$ , where $x_{i}^{(m)} \in R^{d_{m}}$ and $d_{m}$ is the input width of party $m$ , the extractor gives a representation $f_{i}^{(m)} = E^{(m)} (x_{i}^{(m)})$ , where $f_{i}^{(m)} \in R^{d}$ and $d$ is the shared output width, for example $d = 20$ . Here $θ^{(m)}$ lists all weights and biases of that small network, $d_{m}$ counts raw columns at party $m$ , and $d$ counts numbers in every emitted vector.

An embedding or representation (learned summary vector that leaves the extractor) is that output vector $f_{i}^{(m)}$ . It is not the raw bank or hospital columns. It is the learned summary that the network thinks will help later steps. Different parties start with different $d_{m}$ — one may read 40 columns, another 60, another 50 — but they all emit the same $d$ so the server can process them together.

Extractor turns local rows into shared-width vectors. For any unaligned row $x \in U^{(m)}$ , the working form is:

$f = E^{(m)} (x; θ^{(m)})$

where $x \in R^{d_{m}}$ is the raw local row, $θ^{(m)}$ holds the extractor weights, and $f \in R^{d}$ is the feature vector in the shared width. The lecture line was: give unaligned data to the extractor, the extractor is a neural network, the network gives features. The update signal for $θ^{(m)}$ comes from prototype distances, detailed in 14.4, not from labels.

Why train on unaligned data at all. Aligned data are few — 20 in the running example — while unaligned data are many. If we trained only on 20 labeled rows, the extractor would stay weak and would memorise those rows. By training on the large pool of unlabeled local rows with a prototype-distance loss, the extractor learns the shape of local data before it ever sees an aligned row. Then, once it is strong, we pass aligned rows through this improved extractor to get aligned embeddings for the server. In symbols, the improved map gives $h_{i}^{(m)} = E_{im p r o v e d}^{(m)} (x_{i}^{(m)})$ for $i \in A$ , where $h_{i}^{(m)} \in R^{d}$ is the aligned embedding sent to the active party.

Scope: local training fits any party with a large unlabeled pool and a small aligned slice. Assumption: unaligned rows come from the same local process as aligned rows, so shapes learned on $U^{(m)}$ transfer to $A$ . When the two pools drift apart, the transfer weakens.

Picture each extractor as a funnel. Wide mouth at the top takes $d_{m}$ raw numbers — 40 money numbers, or 60 health numbers. Narrow neck at the bottom lets out $d = 20$ numbers. Training squeezes the funnel so that rows from the same hidden class leave through nearby neck positions, even though no label ever names the class. Takeaway: width is fixed by the neck, meaning is shaped by the distance loss.

Traps: (1) sending raw rows to the server — only fixed-width vectors travel; (2) training the extractor only on 20 aligned rows — too few, it would overfit; (3) expecting output widths to differ — code fixes them to one shared $d$ so vectors stack at $S$ .

Recap: $E^{(m)}$ maps $d_{m}$ raw columns to $d$ shared numbers, trained on the large $U^{(m)}$ pool without labels. The handoff: the next step spells out the repeat-until-steady loop that tunes $θ^{(m)}$ .

Clinics that hold thousands of unlabeled visit records but share only a few linked patients with a bank use the same move: shape the local funnel on the many, then emit vectors for the few.

14.2.2 Training Loop and Stopping Rule

How does a network improve when no one tells it the right answer? By asking a softer question on every batch: does each vector sit near some reference point, and does each reference point sit among vectors?

This concept is a procedure, so it is read as inputs, steps, trace and cost rather than as a single formula.

Purpose and inputs. The loop turns a weak $E^{(m)}$ into an improved one using only $U^{(m)}$ . Inputs: a batch of unaligned rows $x \in U^{(m)}$ , current weights $θ_{t}^{(m)}$ at round $t$ , current mixed prototypes $μ ~_{k}^{(m)}$ and mixed priors $π ~_{k}^{(m)}$ . Outputs: updated weights $θ_{t + 1}^{(m)}$ and, at the end, aligned embeddings $h_{i}^{(m)}$ for $i \in A$ .

Steps of one round. Step one, forward pass. Take a batch of unaligned rows $x \in U^{(m)}$ and compute $f = E^{(m)} (x; θ_{t}^{(m)})$ . Step two, loss build. Compare each $f$ to class prototypes using local and mixed global information, the full two-way loss in 14.4, to get a scalar loss $L_{l oc a l}^{(m)}$ . Step three, update. Move parameters against the gradient with step size $η$ , where $η \in R$ is the learning rate, how big each step is. In symbols:

$θ_{t + 1}^{(m)} = θ_{t}^{(m)} - η \nabla_{θ^{(m)}} L_{l oc a l}^{(m)}$

where $\nabla_{θ^{(m)}} L_{l oc a l}^{(m)}$ is the vector of partial slopes of the loss with respect to each weight. The lecture line was: compute the loss in the mixed-probability format, update the extractor neural network, keep moving, stop when there is no change in parameters.

Trace it on a tiny batch to fix the order. Say party 2 draws 32 unaligned rows, each 1 by 60. The extractor emits 32 vectors, each 1 by 20. Distances to three mixed prototypes give a batch loss, say 84.5. The gradient step with $η = 0.01$ nudges each weight by one hundredth of its slope. Next round the same check gives 81.2, then 78.9. The drop is the only report card, since no label exists to score right or wrong.

Cost stays local. One round touches one batch, one forward pass plus one backward pass through a small network. No cross-party message is needed inside the loop except the slow refresh of global prototypes and priors from $S$ . That keeps the heavy work at the edge.

Scope: the loop runs per party, on its own $U^{(m)}$ , at its own pace. Assumption: the mixed prototypes and priors sent back from $S$ are fresh enough to steer local steps. When feedback stalls, local training still runs but loses the federation-wide view.

See the loop as practice swings before the real pitch. Unaligned rows are practice swings with no umpire. Aligned rows are the real pitches the server judges. You want many practice swings so the swing is steady before it counts. The stopping rule matches the image: when the swing stops changing, practice ends.

Do not: (1) stop after a fixed single pass — stop when $∥ θ_{t + 1}^{(m)} - θ_{t}^{(m)} ∥$ stays near zero across rounds or the loss stops dropping; (2) send aligned rows through a half-trained extractor — only the improved $E^{(m)}$ emits $h_{i}^{(m)}$ ; (3) mix up $f$ and $h$ — $f$ is any local output, $h_{i}^{(m)}$ is the aligned embedding for $i \in A$ that travels to $S$ .

The stopping rule is simple. When $∥ θ_{t + 1}^{(m)} - θ_{t}^{(m)} ∥$ stays near zero across rounds, or the loss stops dropping, training halts. The extractor is then called improved or ready. Only then do aligned rows go through it: $h_{i}^{(m)} = E_{im p r o v e d}^{(m)} (x_{i}^{(m)})$ for $i \in A$ , where $h_{i}^{(m)} \in R^{d}$ is the aligned embedding sent to the active party.

Exam note: be ready to write the three moves in order — forward on unaligned batch, distance loss with local plus global priors, gradient update — and to state the stop test as no change in parameters. Name shapes: 1 by 60 in, 1 by 20 out in the running sketch.

Recap: repeat forward, distance loss and gradient step on $U^{(m)}$ until weights steady, then emit $h_{i}^{(m)}$ for the aligned slice. Bridge: the loss used in step two needs reference points and class weights, which is what priors and prototypes supply next.

14.2.3 Student Questions and Answers

Q: If labels are only at the server for aligned data, what loss trains the local extractor on unlabeled unaligned data?

A: It is not a supervised loss with right answers. It is a distance loss between extracted features and class prototypes that mix local averages with global averages sent back from the server. Each feature vector should sit near one prototype blend, and each prototype blend should sit among feature vectors. The gap between vectors and blends is the training signal. When that gap between features and prototypes shrinks, with local averages and global averages both in the blend, the extractor is improving, even though no label was ever seen locally.

Students often expect a hidden label trick. There is none. The trick is geometric: pull each vector toward its nearest blend and each blend toward its nearby vectors, and repeat. The two directions together shape clusters without naming them. Labels only return at the top, when $C$ reads fused aligned rows.

14.3 Local Priors, Global Priors and Mixed Class Probabilities

14.3.1 Local Prior and Global Prior

Your clinic sees mostly cat cases. The federation as a whole sees cats, dogs and elephants in balance. When you judge one new scan, should you trust your clinic rate, the federation rate, or a mix of both?

A prior (probability list over classes before we look at one specific vector) answers how common each class is in general. We write the local prior of party $m$ as $π^{(m)}$ , where $π^{(m)} \in [0, 1]^{K}$ , $\sum_{k = 1}^{K} π_{k}^{(m)} = 1$ , and $K = 3$ for cat, dog, elephant. Entry $π_{k}^{(m)}$ answers: within party $m$ , how common is class $k$ . A bag with 70 red, 20 blue and 10 green marbles is a fair picture: each share is one entry of $π^{(m)}$ .

A global prior (same list but pooled across parties at the server) is written as $π^{g}$ , where $π^{g} \in [0, 1]^{K}$ and entries sum to 1. The server builds $π^{g}$ from the embeddings it receives from all parties, so it reflects the joint view. Entry $π_{k}^{g}$ answers: across the whole federation, how common is class $k$ .

Mixed prior blends the local list with the pooled list. We write the mixed prior of party $m$ for class $k$ as:

$π ~_{k}^{(m)} = α π_{k}^{(m)} + (1 - α) π_{k}^{g}$

where $α \in [0, 1]$ is a mixing weight, how much we trust the local list versus the pooled list. The lecture asked us to focus on the local prior, what is the probability spread of my classes, then get the probable spread from the other clients, and mix local probabilities and global prior. The convex blend above is that mix in symbols: when $α = 1$ we use only local counts, when $α = 0$ we use only pooled counts, and in practice the blend sits between, so a party with skewed local counts still hears the wider picture. The blend stays a valid probability list because a weighted average of two lists that each sum to 1 also sums to 1, which can be checked by adding entries: $\sum_{k} π ~_{k}^{(m)} = α \cdot 1 + (1 - α) \cdot 1 = 1$ .

Work a tiny mix. Say party 1 sees mostly cats: $π^{(1)} = [0.70, 0.20, 0.10]$ . The federation is balanced: $π^{g} = [0.34, 0.33, 0.33]$ . With $α = 0.5$ , the mixed list is $[0.52, 0.265, 0.215]$ . The local 70 percent for cats softens to 52 percent. That softening is the point: the party keeps its own signal but no longer rules out dogs.

A prototype or centroid (average vector for one class) is written for the local view as $μ_{k}^{(m)} \in R^{d}$ , and for the federation view as $μ_{k}^{g} \in R^{d}$ . The mixed or combined prototype used in the loss is $μ ~_{k}^{(m)}$ , a blend of $μ_{k}^{(m)}$ and $μ_{k}^{g}$ . Think of $μ_{k}^{(m)}$ as the party view of a typical cat vector, $μ_{k}^{g}$ as the federation view of a typical cat vector, and $μ ~_{k}^{(m)}$ as the agreed reference point for cat when party $m$ trains.

Scope: mixing helps when local class rates are skewed but the federation rate is steadier. Assumption: $π^{g}$ sent back from $S$ is built from enough fused views to be trusted. When feedback is stale, raise $α$ and lean local until fresh global numbers arrive.

Picture each prior as a bar with three segments for cat, dog, elephant. The local bar of party 3 with only 2 samples might show one huge block and two slivers, a noisy picture. The global bar shows three even blocks. The mixed bar sits between: still tilted local, but no class vanishes. Takeaway: mixing keeps rare classes alive in the weights even when a party barely saw them.

Traps: (1) reading a prior as a verdict on one vector — it is a base rate before seeing the vector, the vector evidence comes next; (2) fixing $α$ at 1 and ignoring the federation — skewed parties then stay skewed; (3) letting vectors of width $d = 20$ drift in meaning — $d$ is the shared width fixed in code, so every $μ$ lives in the same $R^{d}$ .

Recap: $π^{(m)}$ is the party rate, $π^{g}$ is the pooled rate, $π ~^{(m)}$ is their blend that steers soft assignments. Bridge: with class weights set, the next step scores each vector against each reference point.

Hospitals with rare-disease skew live this daily: a small clinic may see zero cases of one illness, yet the federation rate reminds it that the class exists. The mixed prior carries that reminder into every local update.

14.3.2 Similarity Between Features and Prototypes

Two photos of cats can look far apart in pixels yet sit near each other in learned space. How do we score nearness between one vector and one class reference?

Similarity here means how near one feature vector is to one prototype. We write a feature as $f \in R^{d}$ and a prototype as $μ_{k} \in R^{d}$ , where $d$ is the shared width 20 in the session. The lecture gave similarity numbers without naming one norm, and both cosine and negative distance fit that talk. The working definition used through the rest of the lecture is negative squared distance:

$s_{k} = - ∥ f - μ ~_{k}^{(m)} ∥_{2}^{2}$

where $∥ v ∥_{2}$ is the length of vector $v$ , square root of sum of squared entries, and larger $s_{k}$ means nearer. In words: take the gap vector $f - μ ~_{k}^{(m)}$ , square each entry, add them, and flip the sign, so a small gap gives a score near zero and a large gap gives a large negative score. Cosine scoring, which looks at angle rather than gap length, would rank neighbours in a similar way here; the distance reading is kept because the later loss speaks in distances 100, 150 and 200, and distance and negative-distance similarity move in lockstep.

Similarity ranking decides the nearest blend. For one feature compared to three prototypes, the session gave similarities $s_{1} = 0.8$ for class 1, $s_{2} = 0.2$ for class 2 and $s_{3} = 0.5$ for class 3. On similarity alone, the vector looks nearest class 1, since $0.8 > 0.5 > 0.2$ . The lecture line was: compare feature with class prototypes, feature vector and centroid vector, got similarities 0.8, 0.2, 0.5, so this feature looks like class 1. Coordinates $1, 0$ and $0, 1$ named in the walkthrough are a toy 2D sketch on the board to show two prototype spots at right angles, not true 20D vectors, which cannot be drawn. The sketch helps the eye; the 20-number vectors do the work.

Worked read of 0.8, 0.2, 0.5. List the three similarities of feature $f$ to three prototypes: class 1 scores 0.8, class 2 scores 0.2, class 3 scores 0.5. Rank them: first is class 1 at 0.8, second is class 3 at 0.5, third is class 2 at 0.2. Margin between best and next is $0.8 - 0.5 = 0.3$ . Verdict on geometry alone: class 1. Sense-check: the order $s_{1} > s_{3} > s_{2}$ matches the claim that the feature looks like class 1, and the 0.3 margin says the lead is real but not overwhelming, which is why the prior mix in the next step can still widen the answer.

The comparison above is similarity in the teaching scale, while the loss later speaks in distances. The two are two sides of one coin: high similarity means small distance. A vector with similarities 0.8, 0.2, 0.5 would show its smallest distance to the class 1 blend, its largest to class 2.

Scope: similarity scoring applies per vector per prototype, inside every batch. Assumption: all vectors and prototypes live in the same $R^{d}$ after the shared-width fix, so subtraction $f - μ ~_{k}^{(m)}$ is valid. Without matched widths, the gap has no meaning.

Draw one dot for $f$ and three crosses for the three prototypes on a page. Label the dot-to-cross gaps with 0.8, 0.2, 0.5 as similarity tags, longest tag to the nearest cross. The nearest cross is class 1. Takeaway: similarity turns geometry into a ranked list, and the ranked list feeds the posterior next.

Do not: (1) treat 0.8 as a probability — it is a similarity score, probabilities come after mixing with $π ~$ and normalising; (2) treat the $1, 0$ sketch as real embeddings — real ones have 20 entries; (3) mix up input widths 1 by 40 or 1 by 60 with the shared output width — similarity is scored on outputs only.

Recap: similarity compares one feature to three prototypes; 0.8 beats 0.5 beats 0.2, so geometry votes class 1. Bridge: geometry alone can mislead when base rates are skewed, so the next step folds in the mixed prior.

14.3.3 Expectation Maximization and Mixed Posterior

The nearest prototype says class 1. Your clinic base rate plus the federation base rate whisper that class 2 is also plausible. How do we combine nearness with base rates into one honest list?

A posterior (chance per class after seeing one vector) answers: given this one vector, what is the chance it came from each class. We write it as $p (k ∣ f)$ , where $k \in {1, 2, 3}$ , $p (k ∣ f) \in [0, 1]$ , $\sum_{k} p (k ∣ f) = 1$ . A soft assignment (share of one vector given to each class) is written $q_{j k} = p (k ∣ f_{j})$ for vector $j$ .

Improved expectation-maximization cycle in five moves. The session walked through the cycle with an animation rather than dictating closed forms, so the steps below complete that qualitative picture with the standard identities:

$π_{k}^{(m)} = \frac{1}{N} j = 1 \sum N q_{j k}$

where $N$ is the batch size, count of vectors in the batch, and $q_{j k}$ is the soft share of vector $j$ for class $k$ . In plain order: (1) start with current prototypes $μ ~_{k}^{(m)}$ and current priors; (2) expectation move — for each unaligned feature $f_{j}$ , compute $q_{j k} = p (k ∣ f_{j})$ from similarities and the mixed prior $π ~_{k}^{(m)}$ , so vectors near a prototype and from a common class get higher $q$ ; (3) prior refresh — average the $q_{j k}$ across the local batch with the display above to refresh $π_{k}^{(m)}$ ; (4) mix with global — blend $π^{(m)}$ with $π^{g}$ from the server to get $π ~^{(m)}$ ; (5) maximization move — nudge each prototype toward the weighted average of vectors assigned to it, then repeat. The lecture lines were: estimate the local prior, update the expectation maximization step, mix local with global prior, get class 1, class 2, class 3 probabilities with the mixed one.

The punchline in numbers: similarity alone said class 1. After mixing local and global priors, the posterior spread said the vector could be class 1 or class 2. That gap — nearest says one class, prior-mixed says two — is exactly what the loss must then work on. The extractor must shift so that geometry and priors agree more strongly next round. One retelling named class 1 or class 3 instead of class 1 or class 2; either way the point stands that the prior widened the answer beyond the nearest-centroid pick.

Posterior from similarity plus mixed prior with softmax normalising. One natural posterior form that applies the formula to the feature plus mixed prior is:

$p (k ∣ f_{j}) = \frac{π ~ _{k}^{(m)} exp ( s _{j k} )}{\sum _{l = 1}^{K} π ~ _{l}^{(m)} exp ( s _{j l} )}$

where $s_{j k}$ is the similarity of vector $j$ to class $k$ prototype, $π ~_{k}^{(m)}$ is the mixed prior, $exp$ is the exponential that turns scores into positive weights, $K = 3$ is the class count, and the denominator sums over all $K$ classes so the result is a proper probability list that adds to 1. The softmax shape, exponentials over a sum, is what forces the outputs onto the probability scale. A detective updating a hunch as clues arrive is a fair image: the mixed prior is the hunch, the similarity is the fresh clue, the posterior is the revised belief.

Check the form three ways. Shape: $π ~_{k}^{(m)}$ is a scalar, $exp (s_{j k})$ is a scalar, their product over a sum of like products stays a scalar in $[0, 1]$ . Domain: the denominator guarantees entries sum to 1, so the output lives on the probability scale. Spot-check: if $s_{j 1} = 0.8$ , $s_{j 2} = 0.2$ , $s_{j 3} = 0.5$ with even priors, class 1 wins; lift $π ~_{2}^{(m)}$ high enough and class 2 gains share despite weaker similarity, which reproduces the widening seen in the lecture.

How the prior widens 0.8. Start from similarities 0.8, 0.2, 0.5 that favour class 1. Suppose the mixed prior is tilted toward class 2 because the federation sees many dogs, say $π ~^{(m)} = [0.3, 0.5, 0.2]$ . Multiply prior by $exp (s)$ : class 1 gets $0.3 \times exp (0.8) \approx 0.3 \times 2.23 = 0.67$ , class 2 gets $0.5 \times exp (0.2) \approx 0.5 \times 1.22 = 0.61$ , class 3 gets $0.2 \times exp (0.5) \approx 0.2 \times 1.65 = 0.33$ . Total $\approx 1.61$ . Divide: class 1 $\approx 0.42$ , class 2 $\approx 0.38$ , class 3 $\approx 0.20$ . Verdict: class 1 still leads but class 2 is now close, so the posterior reads class 1 or class 2. Sense-check: geometry alone gave a clear lead; the prior pulled the runner-up into contention, which is the widening the lecture stressed.

Scope: the posterior is computed per vector per round, with the current $π ~^{(m)}$ . Assumption: similarities and priors are on compatible scales so their product is meaningful. When a party has only 2 samples, its local prior is noisy and the global part of the mix carries more weight.

See the expectation move as sorting mail into three trays. Each letter $f_{j}$ gets torn into fractions $q_{j 1}, q_{j 2}, q_{j 3}$ across trays, with larger fractions to nearer, more common classes. The prior refresh then weighs each tray. The maximization move slides each tray label $μ ~_{k}^{(m)}$ toward the centre of its torn pieces. Takeaway: soft sorting plus recentring, repeated, sharpens both weights and reference points.

Traps: (1) hardening $q_{j k}$ to 0 or 1 too early — soft shares carry the uncertainty the loss needs; (2) skipping the mix-with-global move — local-only priors lock in skew; (3) reading the posterior as a label — no true $y$ is used, it is a training weight, not an answer key.

Recap: similarities give nearness, the mixed prior gives base rates, the softmax posterior multiplies them into honest shares that can read class 1 or class 2 even when geometry alone says class 1. Bridge: those shares and prototypes now feed a loss that pulls vectors and blends together.

Fraud teams use the same blend: a transaction vector may look nearest the normal cluster, yet a pooled prior that knows fraud spikes on weekends lifts the fraud share enough to keep the case under review.

14.3.4 Student Questions and Answers

Q: Must embedding widths match across clients, even when input widths differ?

A: Yes for outputs, no for inputs. Inputs can be 1 by 40 in one party, 1 by 60 in another, 1 by 50 in a third, or 1 by 512 versus 1 by 256. Those input widths stay different because raw feature counts differ. Outputs must share one width, such as 20 or 256 across all parties, because the server cannot add or stack vectors of different lengths. The code sets the last layer of each extractor to that shared width. Different-size embeddings cannot be processed together, so the shared width is fixed up front. The match rule is: embedding widths match, input widths differ.

Students often ask why the code cannot just pad the short vector. Padding would line up lengths but not meanings: entry 5 of a money vector and entry 5 of a health vector would still mean different things. Shared width is needed, but shared meaning comes later from the adapter and gating, not from padding.

Recap: priors set base rates, similarities score nearness, posteriors combine them. The section hands a complete steering signal — mixed prototypes plus mixed priors plus soft shares — to the two-way loss next.

14.4 Two-Way Prototype Loss

14.4.1 Feature to Prototype Cost

Every guest must find a table. If one guest stands alone in a corner, the seating plan fails. How do we score whether every vector found a home prototype?

The feature-to-prototype cost, called F2mu in the walkthrough, asks: does every feature sit near at least one prototype blend. We write the set of unaligned features of party $m$ as ${f_{j}}_{j = 1}^{N}$ , where $f_{j} \in R^{d}$ , $N$ is the local batch size, count of vectors in the batch, and the mixed prototypes as ${μ ~_{k}^{(m)}}_{k = 1}^{K}$ , where $K = 3$ is the class count and $d$ is the shared width.

Feature to prototype averages each vector's minimum distance. For one feature $f_{j}$ , compute its distance to each blend, $d_{j k} = ∥ f_{j} - μ ~_{k}^{(m)} ∥_{2}$ , where $∥ \cdot ∥_{2}$ is Euclidean length. Take the smallest across $k$ . Across the batch:

$L_{F \to μ}^{(m)} = \frac{1}{N} j = 1 \sum N k min ∥ f_{j} - μ ~_{k}^{(m)} ∥_{2}^{2}$

where $L_{F \to μ}^{(m)} \in R$ is a scalar cost, $F \to μ$ means feature to prototype, and the average divides by $N$ so batch size does not inflate the score. The lecture line was: when you have a representation, how much is the distance, it has to be at least nearer to one centroid, that minimum distance must be small. A second phrasing was: expected cost of moving a feature to prototypes, assigning a feature to mu. The min form above matches the 100 versus 150 versus 200 story told in the session: each vector is assigned to its nearest blend and only that smallest gap counts.

Reading 100, 150, 200. Take one feature $f$ with distances 100 to blend 1, 150 to blend 2 and 200 to blend 3 in the teaching units. Step one, assign: nearest prototype assignment picks blend 1 since $100 < 150 < 200$ . Step two, score: the cost for this vector is $10 0^{2} = 10000$ in squared units, or 100 in root units. Step three, judge: 100 is still large, so assignment alone is not enough and the large gap must fall. Training must pull the feature nearer its nearest blend or move the blend nearer the feature. Sense-check: the nearest tag is right, but the distance value says learning is far from done.

Average that logic over the batch. With $N = 32$ vectors, add the 32 minima and divide by 32. A tight batch might average 15; a scattered batch might average 120. The number is the report card when no label exists: we do not know the true class, but we know each vector came from one of the classes, so its nearest-blend gap must shrink.

Scope: this direction guards vectors — no vector left stranded. Assumption: every unaligned vector belongs to one of the $K$ classes, so a nearest blend always exists. When stray inputs from outside the task enter the pool, they inflate the average and should be filtered.

Draw one dot and three crosses. Draw three dashed lines from the dot, label them 100, 150, 200. Circle the 100 line: that is the only one that counts for this dot. Repeat for every dot and average the circled lengths. Takeaway: each vector pays only its shortest link, and the batch pays the mean of those links.

Traps: (1) averaging all three distances per vector — only the minimum counts, or common classes would always dominate; (2) reading a small minimum as done — 100 assigned correctly is still 100 away; (3) forgetting the square — the loss squares the gap so large gaps pull harder than small ones.

Recap: $L_{F \to μ}^{(m)}$ pulls each feature to its nearest blend and averages those minima over the batch. Bridge: this alone lets all vectors crowd one popular blend, so the reverse direction must guard the blends themselves.

14.4.2 Prototype to Feature Cost

Every table must get guests. If the rare-disease table sits empty while everyone crowds the common-cold table, the plan fails even though every guest sits somewhere. How do we score empty tables?

The prototype-to-feature cost, the reverse direction, asks: does every prototype sit among features, with no prototype left alone. We write it as $L_{μ \to F}^{(m)}$ .

Prototype to feature averages each blend's nearest-vector gap. For one prototype $μ ~_{k}^{(m)}$ , look at all features and take the nearest:

$L_{μ \to F}^{(m)} = \frac{1}{K} k = 1 \sum K j min ∥ μ ~_{k}^{(m)} - f_{j} ∥_{2}^{2}$

where the average divides by $K$ so class count does not inflate the score. The lecture line was: when you have a centroid, the centroid must be surrounded by representations, reverse expected cost of moving prototypes to features. Rare classes keep a voice because their prototype must attract nearby vectors: when the reverse gap for a rare blend stays large, the loss pushes the blend toward its scattered members and pulls those members closer.

Why both directions. One direction alone lets the model cheat. If we only push features toward prototypes, all features could crowd near one common prototype and leave rare-class prototypes empty. The reverse term stops that. Because every class owns a prototype and every prototype must attract nearby features, rare classes keep a voice. The walkthrough stressed this guard: it prevents the model from ignoring rare cases or rare classes. Think of it like seating plans. Feature-to-prototype says every guest finds a table. Prototype-to-feature says every table gets guests. You need both, or one table stays empty while all guests crowd one corner.

Side-by-side contrast helps memory:

Direction	Averages over	Asks	Failure it punishes
Feature to prototype $L_{F \to μ}^{(m)}$	$N$ features	each vector finds a home blend	stranded vectors far from every blend
Prototype to feature $L_{μ \to F}^{(m)}$	$K$ prototypes	each blend attracts nearby vectors	empty rare-class blend ignored by all vectors

End with when to blame which: a large first term means vectors are scattered; a large second term means a prototype is stranded.

Scope: this direction guards prototypes — no class left without members. Assumption: $K$ matches the true class count, here 3. When $K$ is set too high, spare prototypes will always look stranded and the term misleads.

Draw the same dots and crosses, but now circle per cross. For the rare-class cross with only two nearby dots, the nearest-dots gap is what counts. If that gap is wide, the cross must move. Takeaway: the reverse pass looks at the picture from the tables, not the guests.

Do not: (1) drop the reverse term to save time — rare prototypes then drift and die; (2) average the reverse over $N$ — it averages over $K$ blends; (3) confuse direction with value — the two numbers match only in symmetric layouts, in general they differ.

Recap: $L_{μ \to F}^{(m)}$ keeps every prototype surrounded, which is the rare-class guard. Bridge: the two halves now add into one training signal.

Cancer screening gives the stakes: common negatives could crowd one prototype while a rare positive prototype starves. The reverse cost forces the model to keep the rare table set.

14.4.3 Total Local Loss and Extractor Update

Two report cards, one network. How do we turn guest-side and table-side scores into a single step for the weights?

Total local loss adds both directions. The total local loss (single scalar that trains one party extractor) is:

$L_{l oc a l}^{(m)} = L_{F \to μ}^{(m)} + L_{μ \to F}^{(m)}$

where $L_{l oc a l}^{(m)} \in R$ is the scalar cost, $F \to μ$ is feature-to-prototype, $μ \to F$ is prototype-to-feature, $m$ is party index, $K$ is class count, $N$ is local batch size. The lecture line was: add both losses, that is my local loss, I use it for training, I use it to update my feature extractor. The extractor update is the gradient step from 14.2 with this loss:

$θ_{t + 1}^{(m)} = θ_{t}^{(m)} - η \nabla_{θ^{(m)}} L_{l oc a l}^{(m)}$

where $θ_{t}^{(m)}$ lists extractor weights at round $t$ and $η$ is the step size. All symbols here were named on first use in 14.2 through 14.4 and keep the same meaning.

One batch through the sum. Suppose a batch of $N = 3$ gives minima 100, 60 and 20 to nearest blends. Then $L_{F \to μ}^{(m)} = (10 0^{2} + 6 0^{2} + 2 0^{2}) /3 = (10000 + 3600 + 400) /3 = 14000/3 \approx 4667$ . Suppose the $K = 3$ prototypes have nearest-vector gaps 40, 50 and 90. Then $L_{μ \to F}^{(m)} = (1600 + 2500 + 8100) /3 = 12200/3 \approx 4067$ . Total $L_{l oc a l}^{(m)} \approx 8734$ . The gradient step then nudges $θ^{(m)}$ to lower that sum. Sense-check: both halves are large, so both vectors and blends must move; a sum near zero would mean tight clusters with no stranded side.

Contrast pair for memory: supervised top loss uses labels and cross-entropy because the server knows $y$ . Unsupervised bottom loss uses distances because clients see no $y$ . Small spread of distances means tight learning. Large spread means scattered learning that still needs work. The two losses live on different floors and never mix: bottom loss tunes $E^{(m)}$ , top loss tunes $C$ plus adapter plus gating.

Scope: the sum trains one party at a time on its own batch. Assumption: the two halves are on the same squared-distance scale so plain adding is fair. When scales differ across parties, keep the sum inside each party and let gating handle trust at the top.

See the update as tuning both guests and tables at once. The gradient pulls each guest toward its table and each table toward its guests in one move. Over rounds the room settles: guests cluster, tables centre, the sum falls. Takeaway: one scalar, two pulls, repeated until steady.

Pitfalls: (1) weighting one half to zero — the guard disappears; (2) reading the sum without splitting it — always ask which half is large before acting; (3) updating the extractor on aligned rows with this loss — aligned rows are for emitting $h_{i}^{(m)}$ , not for distance training.

Exam note: be ready to define F2mu and the reverse term in one line each, then write the sum. A common test asks why the reverse term matters — answer: it keeps rare-class prototypes live, the rare class guard. Quote the seating image: every guest finds a table, every table gets guests.

Recap: $L_{l oc a l}^{(m)}$ is the two-way sum that moves $θ^{(m)}$ without labels. Bridge: once each party extractor steadies, its aligned vectors travel up to the adapter and gating stage.

14.4.4 Student Questions and Answers

Q: The two losses sound the same — feature to centroid and centroid to feature. Are their values not just equal?

A: No, they differ in direction and in what failure they punish. Feature to prototype averages over features and asks each vector to find a home prototype. Prototype to feature averages over prototypes and asks each prototype to attract nearby vectors. Their numbers match only in special symmetric layouts. In general they differ, and their sum is needed. The reverse term is what keeps a rare-class prototype from being dropped when most vectors crowd the common classes. Direction decides the average, and the rare class guard needs the prototype direction.

A tiny sketch settles it. Put ten dots near cross A and one dot near cross B. Feature-to-prototype averages eleven small gaps and looks happy. Prototype-to-feature averages two gaps, one of which may be large if cross B sits off its lone dot, and flags the problem. Same dots, different verdicts. That is why the exam loves this question.

14.5 Adapter, Gating and Weighted Aggregation at the Active Party

14.5.1 Why Dimensions Can Be Fixed but Distributions Cannot

All three parties emit 20 numbers per vector. Shapes match, code runs, yet the model still stumbles. If widths match, what else can differ?

A dimension here is vector width. We write input width of party $m$ as $d_{m}$ and shared output width as $d$ . The engineer sets the last layer of each $E^{(m)}$ so every $h_{i}^{(m)} \in R^{d}$ . Examples named were 1 by 40, 1 by 60, 1 by 50, 1 by 512, 1 by 256 inputs all mapped to a shared width such as 256 or 20. That part is under control: width mismatch at the input is normal, width mismatch at the output is fixed by design.

A distribution (shape of the cloud of vectors, where values cluster and how wide they spread) is the harder half. Money-based vectors and health-based vectors form different clouds even when widths match, because input qualities differ. Width control cannot fix cloud shape. That is why the server needs an adapter plus learned weights, not just stacking. Stacking 1 by 20 money rows with 1 by 20 health rows lines up columns but mixes meanings: entry 7 means one thing for money, another for health.

Width is engineered, spread is learned. Width matching is a code choice: set each extractor head to emit $d$ numbers. Spread matching is a learning task: map the three clouds into one shared area and learn per-sample trust. The first needs no data; the second needs the aligned slice plus gating training.

Scope: this split matters whenever parties read different column types. Assumption: a small shared-space map exists that keeps class separation while merging clouds. When views are wholly unrelated, no map can join them and fusion adds noise.

Draw three clouds on one page: a tight ball for the 5000-sample party, a loose puff for the 500-sample party, two lonely dots for the 2-sample party. All dots have 20 coordinates, yet the clouds sit apart. The adapter must slide and stretch the puffs onto one patch; gating must then trust the tight ball most. Takeaway: same width, different weather — the server must fix the weather.

Traps: (1) thinking shared width means shared meaning — meaning comes from the adapter; (2) stacking raw vectors without adapting — clouds collide; (3) averaging with fixed equal weights — thin parties get over-trusted.

Recap: dimensions can be fixed in code, distributions cannot and must be learned. Bridge: the adapter learns the map, gating learns the trust.

14.5.2 Adapter Mapping to a Common Space

Dollars, euros and rupees are all money, yet you cannot add the numbers until each is changed to one unit. What is the exchange desk for vectors?

An adapter (small trainable network at the server that maps each party view into one shared area) takes the aligned embedding from party $m$ for aligned sample $i$ , written $h_{i}^{(m)} \in R^{d}$ , and returns the adapted vector $g_{i}^{(m)} = A (h_{i}^{(m)}; ϕ)$ , where $A$ is the adapter with parameters $ϕ$ . After this map, vectors from different clouds sit in comparable coordinates. The lecture line was: they are from different spreads, map them to a common representation, let this be the common spread.

Adapter aligns clouds before any mixing. In compact form $g_{i}^{(m)} = A (h_{i}^{(m)})$ , where $h_{i}^{(m)} \in R^{d}$ is the raw party view and $g_{i}^{(m)} \in R^{d}$ is the shared-space view. The map $A$ is learned at $S$ alongside the classifier, so money-based and health-based vectors land where one fused vector can stand for one aligned sample. Width matching only ensures all bills are paper of the same size; exchange makes values comparable.

Think of it like currency exchange before adding bills. One party hands over dollars, another euros, another rupees. You cannot add them until each is changed to one shared unit. The adapter is the exchange desk. Entry 7 after the desk means the same kind of evidence no matter which party paid it.

Scope: one adapter serves all parties in the teaching design, learning a common spread. Assumption: the aligned slice of 20 rows is enough to fit this small map without overfitting. When aligned rows are even fewer, keep the adapter tiny.

Sketch the desk: three arrows in — 1 by 256 money vector, 1 by 256 health vector, 1 by 256 retail vector — one box $A$ in the middle, three arrows out in one shared patch. Dots that were far apart by source now overlap by class. Takeaway: source colour fades, class colour stays.

Do not: (1) adapt unaligned rows — only aligned $h_{i}^{(m)}$ for $i \in A$ pass the desk; (2) freeze the adapter while training gating — the two learn together; (3) confuse $h$ with $g$ — $h$ is party-flavoured, $g$ is shared-space.

Recap: $A$ turns party-flavoured $h_{i}^{(m)}$ into comparable $g_{i}^{(m)}$ . Bridge: comparable is not yet trusted — gating decides how much each view counts.

14.5.3 Gating Weights and Weighted Sum

Three witnesses describe one suspect. One watched for an hour, one for a minute, one blinked. Should the court weigh them the same?

Gating (learned per-sample trust over parties) learns how much to trust each party for each sample. We write gating weights for aligned sample $i$ as $w_{i}^{(1)}, w_{i}^{(2)}, w_{i}^{(3)}$ , where each $w_{i}^{(m)} \in [0, 1]$ and $\sum_{m} w_{i}^{(m)} = 1$ . Each $w_{i}^{(m)}$ is a scalar trust weight. Weighted aggregation (trust-weighted sum into one vector) builds one fused vector:

$z_{i} = m = 1 \sum 3 w_{i}^{(m)} g_{i}^{(m)}$

where $z_{i} \in R^{d}$ is the fused representation for aligned sample $i$ , $g_{i}^{(m)}$ is the adapted vector, and $w_{i}^{(m)}$ is its scalar trust weight. The weights are learned parameters, not fixed averages. The lecture line was: compute weightages, how much weight for first client representations, second, third, multiply with the scale, add, get a single-width vector.

Weighted sum keeps width while encoding trust. Shape check: if each $g_{i}^{(m)}$ is 1 by 256 and each $w_{i}^{(m)}$ is a scalar, then $w_{i}^{(m)} g_{i}^{(m)}$ stays 1 by 256, and the sum $z_{i}$ stays 1 by 256. One aligned sample present in three parties still yields one fused vector. For 20 aligned samples, the server holds 20 fused vectors $z_{1}$ through $z_{20}$ , arranged as 20 rows. Row one is the fused view of aligned sample one, row two of sample two, and so on. Those rows feed the global classifier.

Trust from sample counts 5000 versus 2. Suppose party 1 trained on 500 cat images, party 2 on 5000, party 3 on 2. Then party 2 vectors carry more learned detail. Learned gating weights might land near $w = 0.80, 0.18, 0.02$ in the teaching sketch — most trust to the rich party, tiny trust to the 2-sample party. Counts map to shares: 5000 of 5502 total is about 91 percent of raw data, yet gating learns 80 percent, not 91, because quality is learned, not copied from counts. Check the sum: $0.80 + 0.18 + 0.02 = 1.00$ , so the mix stays on scale. Sense-check: equal averaging would give each party 33 percent and would over-trust the 2-sample view; learned weights waste less of the strong view.

Weights shift per sample as gating learns. A party weak on cats may be strong on dogs, so $w_{i}^{(m)}$ moves with $i$ . The numbers 80, 18 and 2 percent are one snapshot, with an alternate retelling as 80, 17 and 3 percent — either way the order is the lesson: rich view first, thin view last.

Scope: gating fits tasks with uneven party strength. Assumption: sample count tracks vector quality well enough that learned weights can find it. When a small party holds uniquely key columns, gating must learn to lift it despite few rows.

Draw one fused dot as a tug of three ropes. The thick rope pulls 80 percent toward the rich party view, the middle rope 18 percent, the thin rope 2 percent. The knot lands near the thick-rope end but still feels the others. Takeaway: fusion is a weighted tug, not a vote.

Pitfalls: (1) fixing weights at one third each — thin views get over-trusted; (2) reading a weight as per-class — it is per client view for one aligned sample; (3) sending separate views to the classifier — only the fused $z_{i}$ travels forward.

Recap: gating learns scalars $w_{i}^{(m)}$ and aggregation returns one $z_{i}$ per aligned sample, with quality following training strength. Bridge: those fused rows now meet labels at the classifier.

Credit scoring shows the pattern: a bureau with 5000 linked histories earns 80 percent trust, a new shop with 2 linked baskets earns 2 percent, yet both still contribute.

14.5.4 Student Questions and Answers

Q: Once we mapped to a common embedding, why go back and do weight multiplication? Is the weight per class or per client, and which vector goes to the server?

A: The weight is per client view for one aligned sample, scaled by how informative that view is. Mapping fixes coordinates but not quality. A party whose extractor saw few unaligned samples gives a thin, less sure vector, while a party whose extractor saw thousands gives a rich vector. Gating learns scalars such as 80 percent, 18 percent and 2 percent that reflect that gap, with gating scalars for quality in a weighted sum. What travels forward is the weighted sum, not the separate views: each party adapted vector times its scalar, added across parties, giving one fused vector per aligned sample. The green common-area sketch shows where vectors live. The weighted sum is what the classifier actually reads. The fused vector goes to the classifier at the server.

The confusion is fair: common space sounds done. The missing half is trust. Coordinates tell where a vector sits; weight tells how much to listen. Both are learned at $S$ .

Q: Even the widths might differ across parties, right?

A: Input widths do differ and that is fine. Output widths must match. The code sets every extractor to emit the same length, for example 20 numbers, with code setting the extractor length. Without that shared length, joint processing is not possible. So width mismatch at the input is normal, with input widths differing across parties, while width mismatch at the output is fixed by design, with output widths matching.

Keep the two levels apart in revision: input widths 1 by 40, 1 by 60, 1 by 50 differ by nature; output widths 1 by 20 or 1 by 256 match by code. Adapter handles clouds, gating handles trust, neither can fix a length mismatch — that is fixed before training starts.

Recap: adapter maps spreads to one patch, gating weights quality per sample, aggregation returns one fused row. The next section trains the decision head on those rows and loops back with fresh prototypes.

14.6 Global Classifier Training and Iterative Update Loop

14.6.1 Global Classifier with Labels

Twenty fused rows, each with a right answer. What is the simplest head that turns one fused vector into three class chances, and what loss teaches it?

A global classifier (decision network at the active party) is written as $C$ , with parameters $ψ$ . It reads the fused vector $z_{i} \in R^{d}$ and outputs class scores $y^_{i} = C (z_{i}; ψ)$ , where $y^_{i} \in [0, 1]^{K}$ sums to 1 across $K = 3$ classes. Here $d$ is the shared fused width such as 256 or 20, $K = 3$ counts cat, dog, elephant, and $ψ$ lists the classifier weights.

Global classifier learns from labels on aligned samples. Because the server holds true labels $y_{i}$ for $i \in A$ , it can use a supervised loss. The teaching name was categorical cross-entropy. In symbols, for one aligned sample with one-hot label $y_{i, k} \in {0, 1}$ :

$L_{g l o ba l} = - i \in A \sum k = 1 \sum K y_{i, k} lo g y^_{i, k}$

where $lo g$ is the natural log as used in learning contexts unless stated otherwise, $y_{i, k} = 1$ for the true class and 0 else, and $y^_{i, k}$ is the predicted chance for class $k$ . All symbols: $A$ aligned set, $K$ class count, $C$ classifier, $ψ$ its weights. Only the true-class term survives per row, so each row pays $- lo g$ of its predicted chance for the right class.

One row through cross-entropy. Say aligned sample $A_{1}$ is a dog, so $y = [0, 1, 0]$ , and the classifier predicts $y^= [0.10, 0.75, 0.15]$ . Loss for this row is $- (0 \times lo g 0.10 + 1 \times lo g 0.75 + 0 \times lo g 0.15) = - lo g 0.75 \approx 0.288$ . If the head later predicts $[0.05, 0.90, 0.05]$ , loss falls to $- lo g 0.90 \approx 0.105$ . Verdict: higher chance on the true class means lower loss. Sense-check: a perfect 1.0 gives loss 0, a poor 0.10 gives loss about 2.30, so the scale punishes confident errors hardest.

Concrete numbers first. First pass accuracy 89 was used as the teaching sketch: the classifier gets 89 right out of 100, or 89 percent, and the team asks how to lift it. Nothing at the server alone can lift it beyond better inputs. Accuracy is capped by the quality of $z_{i}$ , which is capped by the quality of local extractors. So the loop must go back to clients with fresh global prototypes and priors, clients improve $E^{(m)}$ , better $h_{i}^{(m)}$ arrive, better $z_{i}$ form, accuracy climbs. Accuracy capped by fused-vector quality is the warning to carry: tuning $C$ alone plateaus near 89 until inputs improve.

Scope: the supervised head fits any small aligned slice with clean labels. Assumption: labels $y_{i}$ at $S$ are correct and cover all $K$ classes. When one class never appears among the 20 rows, the head cannot learn it and the reverse prototype guard from 14.4 matters even more.

Draw the head as a small box $C$ with one arrow in ( $z_{i}$ , 1 by 256) and three arrows out (cat, dog, elephant chances that sum to 1). Colour the true-class arrow: the loss tugs it upward. Takeaway: cross-entropy rewards chance placed on the true class, nowhere else.

Traps: (1) training $C$ on unaligned rows — it only reads fused aligned rows with labels; (2) reading 89 as done — 89 percent first pass is a start that calls for feedback; (3) tuning only $ψ$ when stuck — better $h_{i}^{(m)}$ lifts the ceiling more than more epochs on $C$ .

Recap: $C$ maps $z_{i}$ to three chances and learns with $L_{g l o ba l}$ on the 20 labeled rows. Bridge: the 20-step protocol shows how that learning loops back to clients.

14.6.2 Twenty-Step Protocol and Feedback of Prototypes

Local benches shape vectors. The top table judges them. How does one full round trip between benches and table run, and what travels each way?

This is a procedure, so it is tracked as a numbered loop with messages each way.

Purpose, inputs and outputs of one round. The loop lifts accuracy past the first-pass 89 by cycling improved references. Upward messages: aligned embeddings $h_{i}^{(m)}$ for $i \in A$ . Downward messages: refreshed global prototypes $μ_{k}^{g}$ and global prior $π^{g}$ . Local state: each $θ^{(m)}$ . Top state: adapter parameters $ϕ$ , gating weights $w_{i}^{(m)}$ , classifier weights $ψ$ .

Steps of the 20-step protocol in plain order. The session described a 20-step algorithm that spells out who does what each round:

$clients: U^{(m)} \to f \to L_{l oc a l}^{(m)} \to θ^{(m)} \to h_{i}^{(m)}; server: h_{i}^{(m)} \to g_{i}^{(m)} \to z_{i} \to C \to μ_{k}^{g}, π^{g}$

Unpacked: (1) each client forward-passes unaligned batches through $E^{(m)}$ ; (2) each client computes similarities to mixed prototypes; (3) each client refreshes its local prior by averaging soft assignments; (4) each client mixes local prior with the last global prior $π^{g}$ to get $π ~^{(m)}$ ; (5) each client builds $L_{l oc a l}^{(m)} = L_{F \to μ}^{(m)} + L_{μ \to F}^{(m)}$ and updates $θ^{(m)}$ ; (6) repeat until $θ^{(m)}$ steadies, then forward-pass aligned rows to get $h_{i}^{(m)}$ for $i \in A$ and send to $S$ ; (7) server adapts each $h_{i}^{(m)}$ to $g_{i}^{(m)}$ , gates to $w_{i}^{(m)}$ , fuses to $z_{i}$ ; (8) server trains $C$ on $(z_{i}, y_{i})$ with $L_{g l o ba l}$ , records accuracy; (9) server refreshes global prototypes $μ_{k}^{g}$ and global prior $π^{g}$ from fused views and sends them back; (10) clients plug the new $μ_{k}^{g}, π^{g}$ into their mixed blends and repeat.

Values named in the animation were prototype weights such as $0.5$ and $0.2$ that shift as learning proceeds, then are sent back and shift again. Watch one weight start at 0.5, slide to 0.42 after local steps, return as 0.44 after global refresh: the drift plus steadying is learning made visible. Intraparty imbalance — skewed class counts within one party — is handled because the mixed prior carries the wider spread into local steps, and the reverse loss keeps rare prototypes alive.

Trace the messages for $A_{1}$ . Up: three vectors $h_{1}^{(1)}, h_{1}^{(2)}, h_{1}^{(3)}$ leave the benches. Top: they become $g_{1}^{(1)}, g_{1}^{(2)}, g_{1}^{(3)}$ , then one $z_{1}$ . Down: fresh $μ_{k}^{g}, π^{g}$ return to all benches. Cost per round is small: only aligned vectors up and only prototypes plus priors down, while heavy batches stay local.

Scope: the loop fits label-scarce vertical tasks where aligned rows are few but unaligned pools are large. Assumption: parties stay online for several rounds so feedback lands. When a party drops, its $w_{i}^{(m)}$ should fall and its prototypes age until it returns.

The shared animation shows local training, centroid drift and steadying, aggregation weights updating, prototype refresh, resend. Play it once to watch one full round before reading the equations again: dots cluster, crosses slide to cluster centres, weights tilt toward the rich party, crosses and priors drop back down.

Do not: (1) run a single pass and stop at 89 — the gain comes from cycling; (2) resend raw rows — only $h_{i}^{(m)}$ travel up, only $μ_{k}^{g}, π^{g}$ travel down; (3) skip the mix-with-global step on return — fresh feedback must enter $π ~^{(m)}$ and $μ ~_{k}^{(m)}$ or the next round repeats the last one.

Recap: unaligned batches shape extractors, aligned vectors fuse and train the head, global prototypes and priors return to steer the next round. Bridge: the next step freezes one round into a concrete table shape.

14.6.3 Worked Shape of Aligned Batches

Take aligned sample one, $A_{1}$ . Client 1 sends $h_{1}^{(1)}$ , client 2 sends $h_{1}^{(2)}$ , client 3 sends $h_{1}^{(3)}$ . Same identifier, three views. Server adapts to $g_{1}^{(1)}$ , $g_{1}^{(2)}$ , $g_{1}^{(3)}$ , weights to $w_{1}^{(1)}$ , $w_{1}^{(2)}$ , $w_{1}^{(3)}$ , fuses to $z_{1}$ . That is row one. Repeat for $A_{2}$ to get row two, and so on through $A_{20}$ . The fused table has 20 rows and $d$ columns, for example 20 by 256. That table plus the 20 labels trains $C$ . Network learns, loss is computed, accuracy is read. If accuracy still falls short, the loop resends fresh $μ_{k}^{g}, π^{g}$ and the whole chain improves.

Fused table of 20 rows and 89 percent. Build the table: 20 rows, one per aligned sample $A_{1}$ through $A_{20}$ , each row a fused vector $z_{i} \in R^{256}$ . Pair row $i$ with label $y_{i}$ . Train $C$ with $L_{g l o ba l}$ and read accuracy. First pass accuracy 89 percent means about 89 of 100 equivalent decisions correct in the sketch scale, or roughly 18 of the 20 rows right with borderline misses on the rest. Verdict: the fused table carries enough signal for a strong start but leaves room that only better $h_{i}^{(m)}$ can fill. Sense-check: adding more epochs on $C$ alone moves 89 little; returning fresh prototypes and priors moves it more.

Real life mirrors the table: this fused-table shape is the same pattern used when joining bank, hospital and shop views of one customer into one row for risk or care decisions, without pooling raw columns. Each source keeps its columns; the top table keeps one row per customer plus the label.

Recap: 20 aligned samples give a 20-row fused table that trains the global classifier to 89 percent first pass, with prototype feedback as the path higher. The next section replays the key numbers end to end.

14.7 Worked Numerical Walkthrough

14.7.1 Similarity Numbers and Prior Mixing

Numbers stick better than slogans. Can you replay one vector from raw scores to posterior shares without notes?

Setup in the teaching sketch: party $M$ holds three unlabeled samples. We write them as $U = {u_{1}, u_{2}, u_{3}}$ , where $M$ is client index, $E$ is its extractor, $U$ is unaligned input. Pass them through $E$ to get $f_{1}, f_{2}, f_{3}$ , each in $R^{d}$ , where $d$ is the shared width such as 20.

Geometry votes class 1, priors widen the field. Pick one feature $f$ . Current mixed prototypes are $μ ~_{1}, μ ~_{2}, μ ~_{3}$ . Similarities are $s_{1} = 0.8, s_{2} = 0.2, s_{3} = 0.5$ . Nearest is class 1 since $0.8 > 0.5 > 0.2$ . Local prior estimate from the expectation-maximization averaging plus global mixing gives mixed chances that spread across class 1 and class 2. In posterior symbols with $K = 3$ :

$p (1 ∣ f) + p (2 ∣ f) + p (3 ∣ f) = 1$

with $p (1 ∣ f)$ largest but $p (2 ∣ f)$ non-trivial after mixing, even though $s_{1} > s_{3} > s_{2}$ alone favoured class 1 by a wider margin. That shift is the prior at work. One retelling named class 1 and class 3 as the widened pair; either way the lesson is wider than similarity alone. Hold two readouts side by side: geometry says class 1, prior-mixed posterior says class 1 or class 2. The extractor update must close that gap.

Posterior mixing from similarity alone to mixed shares. Start with similarity alone favouring class 1 at 0.8 versus 0.5 and 0.2. Fold in a mixed prior tilted toward class 2, for example $π ~ = [0.3, 0.5, 0.2]$ . Unnormalised weights are $0.3 \times exp (0.8) \approx 0.67$ , $0.5 \times exp (0.2) \approx 0.61$ , $0.2 \times exp (0.5) \approx 0.33$ . Total $\approx 1.61$ . Normalised posterior mixing gives about $[0.42, 0.38, 0.20]$ for class 1, class 2, class 3. Verdict: similarity alone said class 1 by a clear margin, prior-mixed posterior says class 1 versus class 2 is now close. Sense-check: entries sum to 1.00 and the runner-up rose from a distant 0.2 score to a live 0.38 share, which is the widening to remember.

The detective image helps: the clue (similarity 0.8) points to suspect 1, but the base rate (many class 2 members in the federation) keeps suspect 2 in the frame. The posterior is the revised shortlist, not a final verdict.

Scope: this walkthrough uses one vector to show the mechanism; batches repeat it per vector. Assumption: the mixed prior used here is current, not a stale copy from rounds ago.

Draw a number line for similarities with ticks at 0.2, 0.5, 0.8, then a second line for posteriors with blocks 42, 38, 20 percent. The lead shrinks from 0.3 to 0.04. Takeaway: the prior compresses leads.

Traps: (1) quoting 0.8 as a probability — it is a similarity, the posterior near 0.42 is the probability; (2) hardening to class 1 too soon — the 0.38 share for class 2 must survive into the loss; (3) forgetting the sum-to-1 check — any posterior list must add to 1.

Recap: similarity alone picks class 1, posterior mixing keeps class 1 or class 2 live. Bridge: the loss must now act on that tension.

14.7.2 Distance Numbers and Loss Reading

Similarity tells who is nearest. Distance tells how far. A nearest tag with a huge gap is still a failure — how do we read it?

Distance reading turns gaps into training moves. Suppose for feature $f$ the distances to blends are $∥ f - μ ~_{1} ∥ = 100$ , $∥ f - μ ~_{2} ∥ = 150$ , $∥ f - μ ~_{3} ∥ = 200$ in the teaching units. Assignment picks class 1, but cost 100 is large and must drop. Across the batch, $L_{F \to μ}$ averages those minima. Across prototypes, $L_{μ \to F}$ checks each $μ ~_{k}$ has nearby features. Their sum $L_{l oc a l}$ drives backpropagation that tunes $E$ . In symbols for this vector, the feature-to-prototype part pays $min {10 0^{2}, 15 0^{2}, 20 0^{2}} = 10000$ , and the prototype side separately checks that each blend has some vector near it.

Loss from 100, 150, 200. Step one, assign to nearest prototype: 100 beats 150 and 200, so class 1. Step two, score the vector: minimum distance 100, squared cost 10000. Step three, read the gap: 100 is large, so the vector-blend pair must move together — pull $f$ toward $μ ~_{1}$ and $μ ~_{1}$ toward $f$ . Across three vectors with minima 100, 60 and 20, the batch term is $(10000 + 3600 + 400) /3 \approx 4667$ . Sense-check: assignment is right yet the value says learning is weak; only a falling number across rounds signals real progress.

This looks heavy, but it is three easy pieces. Piece one, each vector finds its nearest blend. Piece two, each blend finds nearby vectors. Piece three, add and step the network. Repeat until the numbers settle. The minimum-distance average is the piece-one score; the reverse minimum is the piece-two score.

Scope: distance reading applies per batch during local rounds. Assumption: teaching units are consistent within a party, so 100 versus 150 comparisons are fair. Across parties with different scales, compare within-party trends, not raw values.

Bar-chart the three gaps: bars at 100, 150, 200. Only the shortest bar is paid, but its height still towers. Next round the bars read 80, 140, 190: same winner, lower bill. Takeaway: watch the height, not just the winner.

Do not: (1) celebrate assignment alone — the large gap must fall; (2) average all three gaps per vector — only the minimum counts; (3) train on aligned rows here — this reading is for unaligned batches.

Recap: 100 wins the assignment but loses on value; the two-way sum must drive it down. Bridge: once local gaps fall, the server numbers take over.

14.7.3 Aggregation Numbers and Accuracy Reading

Local gaps are down. Three views arrive per aligned sample. How do trust weights turn them into one row, and what does 89 mean?

Aggregation turns three views into one fused row. At the server, assume $d = 256$ shared width and gating weights $w^{(1)} = 0.80, w^{(2)} = 0.18, w^{(3)} = 0.02$ for one aligned sample. Then:

$z = 0.80 g^{(1)} + 0.18 g^{(2)} + 0.02 g^{(3)}$

where each $g^{(m)} \in R^{256}$ and $z \in R^{256}$ . Each product $w^{(m)} g^{(m)}$ keeps shape 1 by 256, so the sum stays 1 by 256. If the third party trained on only 2 samples, its 2 percent weight reflects thin evidence. If later it gathers more unaligned data and its extractor strengthens, gating can lift its weight in later rounds because weights are learned, not fixed.

Aggregation with 80, 18, 2 percent and 89 percent read. Take one aligned sample with adapted views $g^{(1)}, g^{(2)}, g^{(3)}$ . Weight them: 80 percent of the rich 5000-sample view, 18 percent of the mid 500-sample view, 2 percent of the thin 2-sample view. Add to one $z$ of width 256. Repeat for all 20 aligned samples to get a 20-row fused table. Train the global classifier and read accuracy. Global read: with 20 fused rows and labels, first-pass accuracy 89 percent means 89 of 100 equivalent decisions correct in the sketch scale, a strong start with headroom. To move past 89, the server cannot just tune $C$ . It must request better $h^{(m)}$ by returning updated $μ_{k}^{g}$ and $π^{g}$ . Better inputs give better $z$ , better $z$ gives better accuracy. That feedback is the heart of the method. Sense-check: weights sum to 1.00, fused width stays 256, and the path past 89 runs through clients, not more top epochs.

Sample counts explain the trust order: 5000 versus 500 versus 2 maps to 80 versus 18 versus 2 percent in learned weights. Counts do not set weights by formula; they explain why learned weights land there. A party that later grows from 2 to 2000 samples should see its share rise in later rounds.

Scope: per-sample gating means these numbers are one snapshot; next sample may read 70, 25, 5. Assumption: adapted vectors $g^{(m)}$ already share coordinates via the adapter, so weighting mixes trust, not units.

See the fused row as a recipe: 80 ml rich stock, 18 ml mid stock, 2 ml thin stock. Taste (accuracy 89) is good. To taste better, grow better stock (better extractors), not just stir longer (more classifier epochs). Takeaway: aggregation weights the present, feedback improves the future.

Pitfalls: (1) fixing weights at equal thirds — wastes the 5000-sample view; (2) reading 89 as a ceiling — it is a first pass before feedback; (3) forgetting the 20-row shape — each of the 20 aligned samples gives one fused row.

Recap: 80, 18, 2 percent fuse one 256-wide row per aligned sample; 20 rows train to 89 percent first pass; prototype and prior feedback is the route higher. Bridge: the closing section turns these numbers into an exam plan.

14.8 Syllabus Closure, Exam Format and Study Plan

14.8.1 Exam Format and Question Style

Forty marks, three questions, one skill tested above all: can you use the method on a small setup, not just recite it?

The closing part set the test shape. Total 40 marks across 3 questions, each with sub-parts numbered like 1.1, 1.2 and 3.1, 3.2. Problems are present alongside conceptual checks. Weight counts 40 percent of the course total, so each mark carries real pull. The paper rewards working, not just answers: setup, formula with symbol meanings, substitution, intermediate values, readout, next move.

What scores. Questions test understanding and use, not recall alone. Expect prompts of the form: given a small setup, how would you apply the method, what loss would you write, how would you read the numbers. A strong answer shows the setup, the formula with symbol meanings, substitution of the given numbers, intermediate values, the final readout, and one line on what the result implies for the next training move. Tables that compare vectors or distances side by side score well because they make minima and weights visible.

Practice the fidelity habit: 0.8 stays 0.8, 0.5 stays 0.5, 89 stays 89, widths like 1 by 40 or 1 by 256 stay as given. Copy numbers exactly, then compute. A posterior that sums to 1.00, weights that sum to 1.00, and a fused width that stays 256 are quick self-checks before moving on.

Scope: the whole term from day one to the final session can seed a sub-question, plus the shared step animations. Assumption: model papers with worked solutions show the expected layout — follow that layout.

See the paper as three stories: one on data and extractors (aligned versus unaligned, $E^{(m)}$ , stopping rule), one on the local engine (priors, similarities 0.8, 0.2, 0.5, distances 100, 150, 200, two-way loss), one on the top loop (adapter, gating 80, 18, 2 percent, fused 20-row table, 89 percent, 20-step feedback). Takeaway: each story wants numbers worked, not just named.

Avoid: (1) writing the loss without naming symbols; (2) rounding teaching numbers until the story breaks; (3) skipping the next-move line — state whether to keep training extractors, refresh prototypes, or retune gating.

Exam note: practice writing assumptions in full, showing work in tables where vectors or distances are compared, and keeping number fidelity — 0.8 stays 0.8, 0.5 stays 0.5, 89 stays 89, widths like 1 by 40 or 1 by 256 stay as given. That habit alone lifts marks across all three questions.

14.8.2 What to Study and In What Order

Four algorithms form the assessed set. Three named papers anchor revision: PET-CVT, Proto-VFL including its efficient prototype variant with unaligned data covered here, and FedCBT named as the remaining lighter algorithm to close next time. Problems discussed across the term plus one or two model papers with worked solutions complete the pack.

Study order that follows the lecture flow. A workable order: start with aligned versus unaligned definitions and label placement, then extractor training loop, then priors and posteriors, then the two-way loss with reasons for each direction, then adapter plus gating plus fusion with the weight-quality link, then the global loop with prototype feedback, then the numbers in 14.7 until each step can be reproduced without notes. Finish with timed problem practice: write the loss, plug numbers, read the outcome, state the next move. This order tracks marks and papers: definitions first, engine next, top loop last.

Run each block as recall plus numbers. Definitions: write $A$ and $U^{(m)}$ plus the 500, 5000, 2 sketch. Extractor: write the three moves plus the stop test. Priors: write $π ~ = α π^{(m)} + (1 - α) π^{g}$ and the 0.8, 0.2, 0.5 read. Loss: write $L_{l oc a l} = L_{F \to μ} + L_{μ \to F}$ plus the 100, 150, 200 read and the rare-class guard. Top: write $g = A (h)$ and $z_{i} = \sum_{m} w_{i}^{(m)} g_{i}^{(m)}$ plus the 80, 18, 2 percent read and the 89 percent path higher. Loop: list the 20-step messages plus the 0.5 and 0.2 prototype drift.

Scope: cover day-one to final-session ground plus the shared HTML walkthroughs and the single consolidated watermark summary once posted. Assumption: anything in that span can seed a sub-question, so breadth plus one fully worked paper beats narrow depth.

Picture revision as layers: bottom layer definitions and extractor, middle layer priors plus loss, top layer adapter plus gating plus classifier plus loop, capstone layer timed papers. Each layer must be reproducible cold before adding the next. Takeaway: build up, then test under time.

Do not: (1) memorise Proto-VFL as a name without its unaligned-data variant steps; (2) leave FedCBT unread because it is lighter — light topics still seed sub-parts; (3) skip the HTML animations — they fix the loop order faster than re-reading equations.

Exam note: cover day-one to final-session ground plus the shared HTML walkthroughs and the single consolidated watermark summary once posted. Anything in that span can seed a sub-question. Close with at least one fully worked model paper with solutions, timed.

Recap: 40 marks over 3 questions reward applied working across four algorithms and three papers — PET-CVT, Proto-VFL family, FedCBT — with number fidelity and next-move reasoning. Bridge: the appendices distil this into targets and domain links.

Exam Guidance Summary

Three questions, 40 marks total, 40 percent weight, with sub-parts such as 1.1, 1.2, 3.1, 3.2. Problems plus understanding-and-application checks test whether methods can be used on small setups, not just recalled.

Key high-yield targets: aligned versus unaligned tagging and label placement, with the aligned set as common identifiers and unaligned pools specific to each party; extractor update $θ_{t + 1} = θ_{t} - η \nabla L_{l oc a l}$ trained on extractor representation feature vectors of shared width with the no-change stop rule; local, global and mixed priors $π ~ = α π^{(m)} + (1 - α) π^{g}$ with posterior probability from similarity and mixed prior through softmax normalising; similarities such as 0.8, 0.2, 0.5 and distances such as 100, 150, 200 with nearest prototype assignment and the large gap reading; two-way loss $L_{l oc a l} = L_{F \to μ} + L_{μ \to F}$ with feature prototype cost as minimum distance average over the batch and prototype feature reverse cost as the rare class guard; adapter $g = A (h)$ plus gating $z_{i} = \sum_{m} w_{i}^{(m)} g_{i}^{(m)}$ with weighted aggregation of the fused vector from gating weights and adapted vectors and the sample-count quality link such as 80, 18, 2 percent reflecting counts 5000 versus 2; global classifier with cross entropy over labels on aligned samples $L_{g l o ba l} = - \sum_{i, k} y_{i, k} lo g y^_{i, k}$ and the fused table of 20 rows with 89-percent first-pass reading; the 20-step loop with prototype weights such as 0.5 and 0.2 plus prior feedback and the intraparty imbalance fix.

Study the four assessed algorithms, the three named papers PET-CVT, Proto-VFL family and FedCBT, the HTML step animations showing centroid drift and weight updates, the consolidated watermark summary, and at least one fully worked model paper with solutions. Write assumptions, keep numbers exact, show tables for comparisons. A strong answer states the setup, names each symbol, substitutes the given numbers, shows intermediate values, highlights the final readout, and closes with the next training move.

Key Industry Applications

Bank plus hospital plus shop views of one customer fused into one decision row for credit, care or fraud checks without sharing raw columns. One identifier, three column blocks, one fused vector $z_{i}$ plus the label at the server — risk teams score default, care teams triage, fraud teams flag, all without pooling raw tables.

Output widths fixed in code such as 20 or 256 so 1 by 40, 1 by 60, 1 by 50, 1 by 512 and 1 by 256 inputs can be processed together, while adapter plus gating handle the remaining cloud-shape gap between money, health and retail spreads. Width matching lines up shapes; the adapter maps clouds to a common spread; gating weights trust per sample.

Trust weighting that favours parties with thousands of training rows over parties with a handful, learned per sample rather than fixed. Weights near 80, 18, 2 percent for counts near 5000, 500, 2 show the link: rich extractors earn weight, thin ones are heard softly until they gather more unaligned data and earn more.

Rare-class guard via the reverse prototype term so infrequent but high-stakes cases keep representation, for example uncommon diagnoses or fraud types. Feature-to-prototype seats every guest; prototype-to-feature keeps every table set, so the rare table never goes empty while common tables crowd.

Label-scarce learning where only a small aligned slice such as 20 rows carries labels while large unaligned pools carry the training load. Local extractors shape on the many without labels through prototype distances; the global classifier learns on the few with labels through cross-entropy; prototype and prior feedback cycles lift accuracy past the 89-percent first pass.

DML Lecture 14 notes · Prototype-Based Vertical Federated Learning with Unaligned Data

Distributed Machine Learning· postgraduate· 2026-09-11

Sections Breakdown

1Aligned and Unaligned Data in Vertical Partitioning

Vertical split with aligned overlap and unaligned pools, labels only at active party.

2Local Feature Extractors Trained on Unaligned Data

Small extractors map raw columns to shared width, trained on unaligned pools with distance loss.

3Local Priors, Global Priors and Mixed Class Probabilities

Local and global class rates blend into mixed priors that shape posteriors with similarities.

4Two-Way Prototype Loss

Feature-to-prototype plus prototype-to-feature distances train extractors without labels.

5Adapter, Gating and Weighted Aggregation at the Active Party

Adapter maps clouds to shared space; gating learns per-sample trust and fuses one row.

6Global Classifier Training and Iterative Update Loop

Classifier trains on 20 fused rows to 89 percent; prototypes and priors loop back.

7Worked Numerical Walkthrough

End-to-end numbers: 0.8 similarities, 100 distances, 80-18-2 fusion, 89 accuracy.

8Syllabus Closure, Exam Format and Study Plan

40 marks over 3 questions; study four algorithms and three papers in lecture order.

Postgraduate students in Distributed Machine Learning

Exam Revision Notes

Below is the distilled, exam-ready core. Every entry comes from the full explanation above. Use this section for rapid review; return to the main notes when a point needs more context.

Aligned and Unaligned Data in Vertical Partitioning

Must-know: Aligned means present in every party; labels live only at server S.

\[ A = I^{(1)} \cap I^{(2)} \cap I^{(3)} \ ]

Top pitfall: Calling a two-party overlap aligned in a three-party task.

Self-check: Who holds labels and what makes a sample aligned?

Connects to: Local Feature Extractors Trained on Unaligned Data

Local Feature Extractors Trained on Unaligned Data

Must-know: Forward on unaligned batch, distance loss, gradient update until no change.

\[ \theta^{(m)}_{t+1} = \theta^{(m)}_{t} - \eta \nabla_{\theta^{(m)}} L^{(m)}_{local} \ ]

Top pitfall: Training extractors only on 20 aligned rows.

Self-check: What are the three loop moves and the stop test?

Connects to: Two-Way Prototype Loss

Local Priors, Global Priors and Mixed Class Probabilities

Must-know: Mixed prior blends local and global rates; posterior mixes similarity with prior.

\[ p(k \mid f_{j}) = \tilde{\pi}^{(m)}_{k} \exp(s_{jk}) / \sum_{l} \tilde{\pi}^{(m)}_{l} \exp(s_{jl}) \ ]

Top pitfall: Reading similarity 0.8 as a probability.

Self-check: How does 0.8 similarity become a 0.42 posterior share?

Connects to: Two-Way Prototype Loss

Two-Way Prototype Loss

Must-know: Total loss is the two-way sum; reverse term guards rare classes.

\[ L^{(m)}_{local} = L_{F \to \mu}^{(m)} + L_{\mu \to F}^{(m)} \ ]

Top pitfall: Dropping the reverse term and losing rare prototypes.

Self-check: Why do 100, 150, 200 need both directions?

Connects to: Local Feature Extractors Trained on Unaligned Data

Adapter, Gating and Weighted Aggregation at the Active Party

Must-know: Adapter fixes spread; gating weights quality; sum gives one fused row.

\[ z_{i} = \sum_{m=1}^{3} w^{(m)}_{i} g^{(m)}_{i} \ ]

Top pitfall: Using equal weights for 5000-sample and 2-sample views.

Self-check: Why do 80, 18, 2 percent beat equal thirds?

Connects to: Global Classifier Training and Iterative Update Loop

Global Classifier Training and Iterative Update Loop

Must-know: Cross-entropy on fused rows; accuracy capped by input quality until feedback.

\[ L_{global} = -\sum_{i \in A}\sum_{k=1}^{K} y_{i,k} \log \hat{y}_{i,k} \ ]

Top pitfall: Tuning only the head when stuck at 89 percent.

Self-check: What travels up and what returns down each round?

Connects to: Adapter, Gating and Weighted Aggregation at the Active Party

Worked Numerical Walkthrough

Must-know: Replay 0.8 to posterior, 100 to loss, 80-18-2 to fused row, 89 read.

\[ z = 0.80\, g^{(1)} + 0.18\, g^{(2)} + 0.02\, g^{(3)} \ ]

Top pitfall: Celebrating nearest assignment while the gap stays large.

Self-check: How does 0.8 become 0.42 and 100 become a training move?

Connects to: Local Priors, Global Priors and Mixed Class Probabilities, Two-Way Prototype Loss, Adapter, Gating and Weighted Aggregation at the Active Party

Syllabus Closure, Exam Format and Study Plan

Must-know: 40 marks, 3 questions, applied problems across full term.

Top pitfall: Rounding teaching numbers until the story breaks.

Self-check: What order covers definitions to loop to timed papers?

Previous LectureLecture 13 · Proto-VFL Under Class Imbalance and Cross-View Training

Next LectureLecture 15 · Sample-Level Client Drift Mitigation and FedBSS in Federated Learning

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.