Vertical and Horizontal Federated Learning

This session splits federated learning by how the data table is cut: by rows (horizontal) or by columns (vertical). The new pain is limited overlapping samples — people who do not sit on every client — and the methods that still train a joint model without shipping raw records.

12.1 Client Drift Recap: SCAFFOLD and FedProx

Last time the class asked a blunt question: if every device trains on its own slice of data, why does the shared model get worse instead of better? The short name for that failure is client drift. Today's vertical and horizontal split still uses local training, so those two named fixes stay in the toolkit.

A client is one device or organization that holds its own data and trains a local copy of a model. Client drift is the failure mode where those local copies walk away from a shared global model. After a few local steps, each client points in a different direction, so the average of their updates is a messy compromise.

12.1.1 Two Fixes for Clients That Wander

Picture a hiking group that agreed to meet at one peak. Each hiker looks at the slope under their own boots. If the hills differ, they fan out. The group average of their footsteps is then a path that nobody wanted.

The spoken cause of the wander in the recap was "IID data". In the usual federated-learning story, drift is worse when data are not independent and identically distributed across clients, often written non-IID. IID means each client's rows look like draws from the same joint distribution. Non-IID means each client sees a different mix of people, labels, or features. The recap used the letters IID; the standard client-drift story (and the textbook treatment of FedAvg on skewed clients) is the non-IID case. Keep both: the recap wording, and the fact that mismatched local distributions are what make the average step lie.

SCAFFOLD tries to correct the direction of each local update. Let $w$ be the current global weights that the server sent down. Client $i$ computes a local gradient $g_i(w_i)$ on its own batch. Left alone, that gradient points toward this client's minimum, not the federation's. SCAFFOLD stores two extra vectors, called control variates:

• $c_i$ is client $i$'s running estimate of its own local gradient.
• $c$ is the server's average of those estimates across clients.

The corrected step on the client is

$$w_i\leftarrow w_i-\eta (g_i(w_i)-c_i+c)$$

where $\eta$ (eta) is the local step size. The term $-c_i+c$ is the compass: it subtracts the client's habitual bias and adds back the group's average direction. If every client had the same data, then $c_i\approx c$ and the correction vanishes. If client $i$ is skewed, the correction shrinks the extra pull of that skew.

FedProx attacks the same wander with a penalty rather than a direction correction. Let $F_i(w)$ be client $i$'s local loss. Instead of minimizing $F_i(w)$ alone, the client minimizes

$$h_i(w;w^t)=F_i(w)+\frac{\mu }{2}\Vert w-w^t{\Vert }^2$$

where $w^t$ is the global model received at round $t$, $\Vert \cdot \Vert$ is the Euclidean (straight-line) length of the weight vector, and $\mu >0$ (mu) is the leash strength. The extra term is zero when $w=w^t$ and grows as the local copy walks away. Large $\mu$ keeps the dog close. Small $\mu$ lets it roam. A companion textbook also notes a second FedProx idea: stop local training when the gradient has shrunk by a factor $\gamma$ (gamma-inexact solution), so slow devices can still contribute without a hand-tuned step count. The recap's teaching picture is the leash term.

When to pick which: use SCAFFOLD when you can store and send a control vector and you care about which way clients walk. Use FedProx when you want a simple extra loss term that limits how far they walk. They are not rivals of today's vertical methods; they are last session's answer to drift.

A common beginner trap is to treat "IID" as the cause of drift. Identical local data make local steps agree. Drift shows up when local data disagree. Another trap is to think FedProx deletes local learning. It does not. It only taxes large moves. A third trap is to mix the two pictures: SCAFFOLD does not add $\Vert w-w^t{\Vert }^2$ to the loss; FedProx does not maintain $c_i$.

12.1.2 Why This Recap Matters for Today

Today's topic is a different split of data: vertical and horizontal federated learning, with extra stress on limited overlapping samples. The drift methods above still matter as background. Clients still train locally. Data still stay local. The new pain is not only "my labels look different from yours". The new pain is "we do not even share the same people, and we do not share the same columns".

Banks, clinics, and phone keyboards still train on-device. The recap's compass and leash remain the right tools when the schema is shared and the rows differ. The rest of this note asks what to do when the schema itself is split.

12.2 Horizontal Federated Learning

How do you cut a spreadsheet so that many phones can train one keyboard model without sending the typed words to a central disk? Cut across the rows. Keep every column. That cut is horizontal federated learning.

The answer is no. Horizontal federated learning shares a feature schema, not a user ID.

12.2.1 Same Columns, Different Rows

Horizontal federated learning (HFL) splits data by samples, which are the rows. Every client stores the same feature columns. Client 1 may hold persons 1, 2, and 3. Client 2 may hold other persons. Client 3 may hold still other persons. Feature 1 through feature 6 are the same names on every client: the same measurements, in the same order.

Let the shared feature vector live in ${\mathbb{R}}^d$. For a running six-column demo, $d=6$ and the named columns are $F_1,F_2,\dots ,F_6$. Client $k$ holds a row set ${\mathcal{I}}_k$. Its local table is

$${\mathcal{D}}_k=\{(x_i,y_i){\}}_{i\in {\mathcal{I}}_k},x_i=(F_{1,i},\dots ,F_{6,i})\in {\mathbb{R}}^6$$

Every other client uses the same six names in the same order. The row index sets ${\mathcal{I}}_k$ can be disjoint. A missing person on client 2 is a missing row, not a missing slice of columns.

If a row is present on a client in this setting, that row usually has the full column set. If a person is missing on one client, you train on the rows you have. You do not have to invent another hospital's lab tests for that person.

Companion textbooks also call this sample-based or homogeneous federated learning. A phone keyboard (Gboard-style next-word prediction) is the stock example: every handset stores the same kinds of typing features for a different user.

Visualize a grid. The horizontal axis lists $F_1$ to $F_6$. The vertical axis lists people $P_1,P_2,\dots$. Client A owns the top block of rows, all six columns filled. Client B owns the next block, all six columns filled. The cut is a horizontal line through the people axis. Takeaway: the width of the table is shared; the height is partitioned.

12.2.2 Many Users, Little Forced Overlap

A bank, a retail store, and a third firm can still appear in a horizontal picture if each stores the same feature list for different users. There is no need for the same person to sit in every client. Overlap of people is optional. The shared object is the feature schema, not the user ID.

A phone keyboard app on many devices is the classic horizontal split. Each phone stores the same kinds of typing features for a different user. The federation averages models (or model updates), not raw keystrokes. That is why horizontal federated learning scales to millions of devices: you do not wait for the same person to type on two phones.

A pitfall is to demand overlapping user IDs in a horizontal design. You can average models without matching people. Another pitfall is to call any multi-company project "horizontal". If the bank stores income and the shop stores spend, the columns differ, and that is vertical. A third pitfall is to treat a missing row as a missing feature: in HFL you simply skip that person on that client.

Banks still show up in both pictures. The test is the schema. Same columns, different customers: horizontal. Different columns, some shared customers: vertical.

12.3 Vertical Federated Learning

Now cut the same spreadsheet down the columns. One clinic keeps heart measurements. Another keeps eye measurements. A third keeps skin measurements. Some patients visit more than one clinic. Many do not. That cut is vertical federated learning.

Vertical federated learning is built for that refusal.

12.3.1 Same People, Different Columns

Vertical federated learning (VFL) splits data by features, which are the columns. Clients may share some of the same people, but each client stores a different view of those people.

Look column-wise. Client A might store heart measurements. Client B might store eye measurements. Client C might store skin measurements. The people can overlap. The columns do not have to.

A bank, a retail store, and another firm fit this picture well. The same person can have a bank account and also buy from the store. A few rows overlap. Many rows do not. Each firm keeps its own feature family. Companion textbooks call this feature-based or heterogeneous federated learning, and they use the same bank-plus-shop story: credit patterns on one side, shopping behavior on the other. Finding the shared people can use a private set-intersection step so IDs match without dumping the full customer lists. The methods in later sections assume that alignment is already known.

A hospital group in one city and a head office in another country may refuse to ship raw images across the border. Vertical federated learning is built for that refusal. One site holds one view. The other site holds another view and often the label.

Compare the two cuts on the dimensions that actually differ:

When to pick which: if every party already stores the same measurements for different users, pick horizontal. If parties store different measurements of partly the same users, pick vertical.

12.3.2 Heart, Eye, and Skin Views of One Health Index

A running health example makes the split concrete.

• One client is a heart doctor and stores heart features. Speech-to-text sometimes rendered this as "hot features". The rest of the example is heart, eye, and skin, so heart is the reading that fits.
• One client is an eye doctor and stores eye features.
• One client is a skin doctor and stores skin features.

The health index is a score or class that needs all three views as input. Feed heart, eye, and skin together, and a model can output the health index. Feed only one view, and the model is starved.

Imagine three strips of film of the same street. Overlapping samples are frames where all three cameras clicked at once. Unaligned samples are frames where only one camera was rolling. The film is real. It still cannot make a three-camera movie until you fill or carefully use those single-camera frames.

12.3.3 Student Questions and Answers

If you train only on $P_1$ and $P_2$, those two people dominate the fit. That is both a fact and a trap.

A beginner trap is to treat "vertical" as "more private horizontal". Privacy rules can apply to both. The geometry is what changed: columns, not rows. Another trap is to assume every patient appears in every clinic. Real overlap is often small. Two hospitals in different countries are not expected to share a large patient list. That limited overlap is the bottleneck the rest of this note attacks.

12.4 Overlapping Samples Versus Unaligned Samples

The health-index table had two kinds of people. Some IDs show up in more than one clinic. Some IDs show up in only one. Mixing those two kinds under the word "overlap" is how the rest of the method gets confused.

This session's hard problem is overlapping samples. Feature overlap is a milder, separate cleanup.

12.4.1 Shared IDs and One-Client IDs

An overlapping sample, also called a shared sample or an aligned sample, is a person or entity whose ID is present in more than one client. In notation used in the walkthrough, $O$ or $S$ marks overlap. Samples $S_0,S_1,S_2,S_3$ were the shared rows in the bank-retailer demo later.

An unaligned sample is a person present in only one client. $U$ marks unaligned. Samples $U_0$ and $U_1$ lived on one side only. They have no counterpart on the other client, and the server stores no label for them.

Write the two index sets as

$$O=\{i:\text{ID }i\text{ exists on more than one client}\},U_k=\{i:\text{ID }i\text{ exists only on client }k\}$$

In the clinic table, $O=\{P_1,P_2\}$ and the unaligned people are $P_3,P_4,P_5,P_6$. In the bank-retailer demo, $O=\{S_0,S_1,S_2,S_3\}$ and one side also holds $U_0,U_1$.

Overlap is about rows, not about whether two clinics happen to measure a similar quantity. Say "overlapping samples". Do not say "overlapping features" as if that were the same problem. Feature overlap is a different, milder issue, treated next.

Think of two clubs that share a few members. The overlapping samples are people with two membership cards. Unaligned samples are people with one card. The clubs might both ask "age". That repeated question is a shared column name, not a shared member.

12.4.2 Feature Overlap Is Not the Main Bug

Two clients might both store a quantity that is almost the same, for example a factor that appears in both skin notes and eye notes. You can merge those two columns, average them, drop one, or keep both. That cleanup is an internal project choice. You can talk to a manager and agree which column to keep.

Even if the same named feature has different values on two clients, that is still not the core research problem here. You can reconcile values later, once you are inside the project.

The work under study addresses a harder hole: $P_1$ and $P_2$ have matching rows across clients, but $P_3$ has only one view, $P_4$ has only another view, and $P_5,P_6$ have only the third view. Those people cannot enter a model that needs every view. Their information sits on disk and is unused. Non-overlapping rows lose information.

Values can still disagree after you name the column the same way.

12.4.3 Why Non-Overlap Wastes Data

If you ignore unaligned people and train only on $P_1$ and $P_2$, you have a limited training set. Limited data give weaker accuracy. At the same time you cannot just "leave" the non-overlapping rows. They are real measurements. They cost money and clinical time. The aim is to stop wasting them.

Unaligned does not mean "trash". It means "this row is incomplete across clients". Completing it, or using it in a careful semi-supervised way, is the whole game.

A two-hospital picture from the communication-efficient VFL paper makes the scale concrete: two hospitals in different countries are not expected to share a large patient list. The overlapping set $N_o$ can be hundreds while each side holds thousands of unaligned rows. Training only on $N_o$ leaves most of the film on the cutting-room floor.

12.4.4 Student Questions and Answers

A natural first idea is: fill the blanks with an average, then train. That idea is the wrong tool when the blank is an image.

Pitfalls to keep separate:

• Saying "overlapping features" when you mean overlapping people.
• Treating value mismatches (age 34 vs 35) as the VFL research problem.
• Calling unaligned rows outliers to delete. Some of them are outliers later, but first they are unused measurements.

12.5 Completing Missing Features Without Naive Imputation

The last section left unused people on disk. The goal is not a prettier spreadsheet. The goal is a bigger joint training set that still respects privacy: raw images stay in the clinic; only later representations and gradients move.

Because a missing scan is not a missing number. It is a missing view.

12.5.1 Goal: Turn Unaligned Rows into Usable Rows

The end goal is a bigger joint data set. Overlapped people already have every view. The method must also make $P_3,P_4,P_5,P_6$ usable, as if they were overlapped, so every sample can help train a model of the health index, or of bank-retail risk.

In the bank-retailer picture, the server holds a risk label only for shared people. Bank features alone are not enough for a good risk label. Retail features alone are not enough either. You need both views. Unaligned people have no server label. Completing their missing view, and later a careful pseudo-label, is how they join training.

Verbal goal: fill the holes so the table is complete enough to train. Not "drop the incomplete people". Not "pretend the hole is the column mean".

Write the wish in symbols. Let $x_i^A$ be client A's view of person $i$ and $x_i^B$ be client B's view. For $i\in O$ both views exist. For an unaligned person $u$ on A, $x_u^B$ is missing. The later algorithms either

• use $x_u^A$ locally to enrich representations (one-shot), or
• estimate a stand-in ${\hat{h}}_B^{(u)}$ in representation space (few-shot),

without shipping the raw row.

12.5.2 Images Break Mean and Mode Fill-In

A common first idea is: train on fully available rows, then predict the missing parts. That instinct is in the right family. The catch is the type of hole. These are not a few numeric blanks in a spreadsheet. They can be images. Mean, mode, and other textbook imputations do not handle an image. The methods below learn representations and relations across clients instead of pasting an average number into a blank cell.

Mean fill-in for a numeric age of $35,40,45$ would write $40$ into a blank. Mean fill-in for a $256\times 256$ scan would write one average gray value into every pixel. That gray square is not a heart. Mode fill-in is the same failure with a "most common pixel". The rejected analogy is exactly that spreadsheet trick applied to medical images.

12.5.3 Three Research Methods Chosen for Study

Three methods from strong papers were chosen as the study set for this session and the next:

1. Communication-efficient vertical federated learning with limited overlapping samples, from Duke University authors together with people at NVIDIA. First author: Jingwei Sun. A spoken nickname in the room was the "SUN" algorithm; that name points at this Sun et al. method, not at a fourth paper. This is the method walked in detail today, including its one-shot and few-shot modes.
2. Proto-EVFL (also spoken as Proto-EFL), a prototype-based vertical method. A first pass appears at the end of this note. A deeper pass is for the next session.
3. FAT-CVT, a third algorithm named in the room. In the same literature the published cousin is FedCVT (federated cross-view training): it estimates missing representations and pseudo-labels to expand limited aligned samples, but it still uses many communication rounds. Today's Duke-NVIDIA method cites that line of work and then cuts communication to one or a few shots.

Exam note: you do not need every symbol in these papers. You do need the flow: what is missing, what is sent, what is clustered, what is checked, and what is gained.

Pitfalls: treating mean-imputation as "good enough for images"; thinking the Sun method shares raw scans; mixing FAT-CVT / FedCVT into the one-shot download story. FedCVT still chats every iteration. One-shot VFL downloads gradients once.

12.6 Communication-Efficient VFL Architecture

Vanilla vertical federated learning chats with the server every local step. That is expensive, and it still only trains on the small overlap. Sun et al. (Duke University and NVIDIA) cut the chat to one download, and they put unaligned rows to work.

The packet in this design is a gradient matrix plus the integer class count $C$.

12.6.1 High-Level Pipeline

The Duke-NVIDIA pipeline, taught with a bank-versus-retailer demo, has this shape.

Most of the heavy supervised learning sits on the server. Clients still run clustering, local SSL, and extractor updates. It is a mix, with the server doing the labeled loss.

Steps of the one-shot path, matching the paper's Algorithm 1:

1. Each client holds its own raw table. Hospital data was named as one possible client store. Bank features and retailer features were the running demo.
2. Each client owns a feature extractor, which is a neural net (or the flatten stage of a net) that turns a raw row into a representation vector.
3. For overlapping rows, each client pushes the row through its extractor and sends only the representation to the server.
4. The server concatenates the views, runs the classifier, computes a cross-entropy loss against $Y_o$, and back-propagates to get one gradient vector per overlapping sample for each client's representation block.
5. The server sends each client its gradient matrix for the overlap rows, plus a clue $C$: how many classes exist among those overlap labels. This gradient packet is "the only download of the entire session" in the one-shot story.
6. Each client runs k-means on its gradient rows with $K=C$ and gets pseudo-labels for overlap rows.
7. Each client then runs local semi-supervised learning (local SSL) on overlap plus unaligned rows, which updates the extractor.
8. Clients upload the new overlap representations. The server fine-tunes $f_c$.
9. If the mode flag is few-shot, extra server-side work estimates the missing client's representation for unaligned rows and applies a two-classifier trust check before a new pseudo-label is kept.

The mode flag in the paper is the string few_shot. If that flag is off, the extra block is skipped and the pipeline stays one-shot.

Communication cost is the point. Vanilla VFL and FedCVT may run thousands of rounds on CIFAR-10. One-shot VFL uses two uploads and one download for the training session. Reported numbers in the paper: more than $46.5\%$ accuracy gain and more than $330\times$ less communication versus prior VFL methods on that image split. You do not need those numbers on the exam. You do need the advantage (tiny chat, uses unaligned rows) and the drawback (one-shot still does not assign a trusted label to each unaligned person; unaligned outliers can still tug the extractor).

12.6.2 One-Shot Versus Few-Shot Modes

The same paper has two modes.

One-shot VFL uses unaligned rows only to enrich representations. It does not try to write an explicit label on each unaligned person. Semi-supervised learning pulls unaligned points toward nearby overlap structure.

Few-shot VFL does extra work after local SSL. The client sends unaligned representations to the server. The server estimates the missing view on the other client, combines views, and only then may assign a high-quality pseudo-label if two classifiers agree.

If a mode flag is few-shot, that extra block runs. If it is not few-shot, the rest of the pipeline stays as in one-shot. Standard form in the paper: if mode == "few_shot".

When to pick which: start with one-shot when chat is the bottleneck and you can live without unaligned labels. Switch on few-shot when overlap is so small that the extractor needs more supervised mass, and you can pay one extra round.

12.6.3 Exam Notes

12.7 Representation Extractors, Privacy, and the Server Classifier

The pipeline is abstract until you put numbers on a bank row. This section does that, then states the privacy rule that makes the numbers legal to send.

Because it already holds the overlap labels. Clients send representations, not labels, and not raw tables.

12.7.1 Bank and Retailer Feature Tables

Client A is a bank. It stores features such as age, income, and balance. Overlap rows are $S_0,S_1,S_2,S_3$. Unaligned rows on this side include $U_0$ and $U_1$.

Client B is a retailer. It stores features such as orders, returns, and spend. It has the same overlap IDs $S_0$ through $S_3$, and its own unaligned people.

The server stores a risk label for overlap people only. Risk is something like no-risk versus risk, or a small set of classes $0,1,2$. The server will not invent a label for $U_0$ at the start, because that person is not shared.

12.7.2 Extractors and the 4 by 3 Representation Matrix

A representation is the numeric vector that comes out of the extractor when you feed a raw row. In a deep net this is often the vector after a flatten layer, before the final class head. The property we want is: the representation is useful for classification, but it is not the raw record.

Each overlap row on client A is pushed through extractor $f_A$ with parameters ${\theta }_A$. If there are four overlap rows and each representation has width 3, client A sends a $4\times 3$ matrix. The spoken form was "client A sends its 4 cross 3 representation matrix to the server".

$$h_A^{(S_i)}=f_A(x_A^{(S_i)};{\theta }_A)\in {\mathbb{R}}^3$$

$$H_A^{(O)}=\left[\begin{array}{c}{h}_A^{(S_0)}\\ h_A^{(S_1)}\\ h_A^{(S_2)}\\ h_A^{(S_3)}\end{array}\right]\in {\mathbb{R}}^{4\times 3}$$

where $h_A^{(S_i)}$ is the representation of overlap sample $S_i$ on client A. Client B sends its own overlap matrix $H_B^{(O)}$ of the same four people, from retail features, through extractor $f_B$. The width $3$ matches the classroom "4 cross 3" demo. A real net may emit width $128$ or $512$. The algebra does not change: $N_o$ rows by representation width.

Standard form in Sun et al.: $h_{o,i}^k=f_k(x_{o,i}^k;{\theta }_k)$ and the server concatenates views with $\circ$. Notation note: texts write $H_o^k$; here we keep $H_A^{(O)}$ for the bank side to match the demo.

Verbal description: pass every overlap row through the extractor, collect the outputs, and ship that small matrix. Do not ship the raw table.

On the server the four people become one joint matrix by concatenation along the feature axis:

$$H^{(O)}=[H_A^{(O)}|H_B^{(O)}]\in {\mathbb{R}}^{4\times 6}$$

That $4\times 6$ block is what the server classifier $f_s$ (same role as $f_c$ in the paper) will score against the four labels. Dimensional check: four rows in, four class-score vectors out.

12.7.3 Privacy Rule: Raw Data Stays Local

Federated learning's core privacy rule in this story: real data should not leave the client. Only representations travel up. Later, only gradients travel down. Labels never leave the server.

That rule is why a head office in one country and a client office in another city can cooperate. Neither side dumps its full database on the other. Images stay in the hospital. Risk labels stay with the party that owns them.

This vertical pattern won an internal award in industry practice, because two offices can train a joint model without a full data swap. A later student applied vertical federated learning inside a company and received an appreciation mail for it.

12.7.4 Student Questions and Answers

The first doubt is usually "if I only send a vector, where does ground truth come from?"

Those labels are ordinary class indices.

A third doubt is that the client looks idle.

Pitfalls: uploading raw $35,60,12$; assuming the server must be told the label; assuming the client only infers. Another pitfall is treating width $3$ as a law. It is a chalkboard size.

12.8 Gradient Clustering with K-Means and Pseudo-Labels

The server has labels. The clients do not. The trick is to send something that still carries class structure without naming the class.

Because the labeled loss pulls same-class representations in the same way. That is the crux: same-class gradients point the same way.

12.8.1 Cross-Entropy on Overlapping Labels

The server concatenates the overlap representations from the clients and feeds them to a server classifier $f_s$. For each overlap sample $i\in O$, the classifier outputs a predicted class distribution ${\hat{y}}_i$. The server also holds the true label $y_i$.

The spoken loss is cross-entropy between the server model output and the ground truth. The session called this "the representation" without writing a concat formula. Sun et al. settle it: the classifier sees the concatenation of all clients' representations.

$$h_i=h_A^{(i)}\circ h_B^{(i)}$$

$${\mathcal{L}}_{\text{CE}}=\sum _{i\in O}{\ell }_{\text{CE}}(f_s(h_i),y_i)$$

where $h_i$ is the combined overlap representation, $y_i$ is the true server label, $\circ$ is concatenation, and ${\ell }_{\text{CE}}$ is cross-entropy. Verbal description: compute the cross-entropy between the output of the model of the server and the ground truth.

For a $C$-class problem the per-row term is

$${\ell }_{\text{CE}}(f_s(h_i),y_i)=-\log p_{i,y_i},p_i=\mathrm{softmax}(f_s(h_i))$$

Hidden labels in the four-row demo were given as:

• $S_0$ has true class $0$
• $S_1$ has true class $1$
• $S_2$ has true class $0$
• $S_3$ has true class $2$

So $C=3$ classes among overlap rows. The server knows this. The clients do not receive $y_i$.

12.8.2 One Gradient Vector Per Sample

After the loss, the server back-propagates to the representation of each overlap sample and gets one gradient vector per sample. Verbal description: the server runs the classifier on the representation, measures the loss against the true labels, and back-propagates to get one gradient vector per sample.

$$g_i^A={\nabla }_{h_A^{(i)}}{\mathcal{L}}_{\text{CE}},g_i^B={\nabla }_{h_B^{(i)}}{\mathcal{L}}_{\text{CE}}$$

where $g_i^A$ is the gradient of the labeled loss with respect to client A's representation of sample $i$. Stacking the four overlap rows gives a gradient matrix $G^A\in {\mathbb{R}}^{4\times 3}$ that the server can send back to the bank. Client B gets its own $G^B$. Paper notation: ${\nabla }_{H_o^k}\mathrm{Loss}$.

Labels never leave the server. Gradients do. The hope is that $g_i$ still carries class structure: two people of the same risk class should produce similar gradient arrows, even if the client never saw the class name.

Why the arrows match: if $S_0$ and $S_2$ both have $y=0$, both softmax vectors are pushed toward the same one-hot target. The residual $(p_i-\mathrm{onehot}(y_i))$ has the same pattern, so the back-prop into $h$ has similar sign and similar size.

12.8.3 The Single Download: Gradients Plus Class Count

The server sends each client:

1. The gradient matrix for that client's overlap rows.
2. The integer $C$, the number of classes among overlap labels.

Verbal description: the server sends each client its gradient matrix, and it is telling that $C$ the number of classes. This is the only download of the entire session.

In the demo, $C=3$: class $0$, class $1$, class $2$. The client is told "these gradients live in three groups". The client is not told which group is "risk" versus "no risk".

The client then runs k-means with $K=C$ on the four gradient rows. K-means is the procedure that drops $K$ centre points into the space and assigns each row to the nearest centre, then moves the centres to the mean of their assigned rows, and repeats.

12.8.4 Same-Class Gradients Point the Same Way

This is the crux. Same-class gradients point the same way. If two overlap people share a hidden label, their gradient rows look nearby: similar signs, similar magnitudes. K-means with $K=C$ can recover groups that match the hidden classes, even though the client never received $y_i$.

The client assigns pseudo-labels ${\tilde{y}}_i$ as cluster IDs $0,1,2$. Those IDs may be a permutation of the server IDs in a messy world, but the teaching assumption is that the same structure is used on both sides. If the server calls cat images class "cat", the client also treats cat as cat. There is no mismatch of names.

Spoken nearby numbers used to show "very nearby" pairs included $9$ and $11$, $34$ and $30$, $48$ close to them, and a negative group $-30,-34,-11$. Those values were read off a demo of gradient entries. They are not a full official matrix. Treat them as a chalkboard of "same-class entries sit next to each other".

Unaligned people never took part in the labeled loss.

12.9 Local Semi-Supervised Learning and One-Shot VFL

The client now has cluster IDs for overlap people and raw rows for everyone else. One-shot VFL spends the unaligned people on a richer extractor, not on a published class name.

They sit near the overlap clusters and pull the representation map so those clusters become sharper.

12.9.1 From Pseudo-Labels to a Local Labeled Set

After k-means, the client still does not know the true server labels. It does know which overlap samples belong together. In the demo, $S_0$ and $S_2$ are together, and $S_1$ and $S_3$ are different from that pair and from each other.

The client now has, for each overlap row, the original features $x_i$ plus a pseudo-label ${\tilde{y}}_i$. That pair $(x_i,{\tilde{y}}_i)$ resembles a labeled data set. Verbal description: the client now has something resembling a labeled data set, $x$ comma $y$, but this $y$ is from the cluster predictions.

$${\mathcal{D}}_O=\{(x_i,{\tilde{y}}_i){\}}_{i\in O}$$

where $x_i$ is the client's raw overlap features and ${\tilde{y}}_i$ is the cluster ID from gradients.

The client also still has unaligned raw rows $\{x_u{\}}_{u\in U}$ with no label.

12.9.2 Supervised Loss on Overlap Plus SSL on Unaligned Rows

Instead of waiting for another server update, the client trains its own extractor locally.

On overlap rows it uses a supervised loss: features plus pseudo-labels.

On unaligned rows it uses an unsupervised or semi-supervised loss: the extractor already knows the overlap clusters, so an unaligned point that sits near a cluster can sharpen that cluster's representation.

Verbal description: overlapping samples plus the pseudo-label from gradient k-means; unaligned rows also used; unsupervised loss; update the feature extractor.

The session described the unaligned term as "some logic", not a named equation. Sun et al. fill that gap. The local objective is

$$l_{\mathrm{ssl}}({\theta }_k;X_u^k,X_o^k,{\hat{Y}}_o^k)=l_s({\theta }_k;X_o^k,{\hat{Y}}_o^k)+{\lambda }_u{l}_u({\theta }_k;X_u^k)$$

which in classroom notation is

$${\mathcal{L}}_{\text{local}}={\mathcal{L}}_{\text{sup}}({\mathcal{D}}_O)+{\mathcal{L}}_{\text{SSL}}(\{x_u{\}}_{u\in U})$$

where ${\mathcal{L}}_{\text{sup}}$ (same role as $l_s$) is supervised loss on pseudo-labeled overlap, ${\mathcal{L}}_{\text{SSL}}$ (same role as $l_u$) is the semi-supervised term on unaligned rows, and ${\lambda }_u\ge 0$ balances the two. For images the paper uses FixMatch: a weakly changed copy and a strongly changed copy of the same unlabeled row should get the same class. For tables they mask features and add noise (FixMatch-tab). Any such recipe is allowed as a local implementation.

Semi-supervised learning (SSL) means a few points have labels and many do not. You feed both into the model. The model may emit pseudo-labels for unlabeled points, then train again on the union. The teaching picture is: train on overlap, then bring in an unaligned row, read which cluster it is near, and let that row slightly change the extractor so representations become richer.

The data that used to be wasted now helps. Accuracy comes from that extra mass, even before you dare to write a hard class name on each unaligned person.

12.9.3 What One-Shot Does and Does Not Predict

In one-shot VFL, you are not predicting an explicit label for each unaligned sample. You use unaligned rows only to improve the extractor. The assumption is that an unaligned sample is closer to one of the known overlap groups. Internally, SSL may learn a soft assignment. The method is not trying to publish that assignment as a trusted label.

After this local training, improved overlap representations can be sent back toward the server for another labeled pass. Unaligned rows in one-shot stay a local sharpening tool.

That is why the mode is called one-shot: two uploads (representations before and after SSL) and one download (gradients plus $C$). No extra round to name $U_0$.

12.9.4 Risks: Outliers, Drift, and Clustering Limits

Because unaligned rows have no class attached, they may be outliers. They can pull representations the wrong way and bring drift. That drawback was accepted in the room as real. Remember the foggy-hill hiker from the recap: an unlabeled point far from every trail can yank the boots off the path. One-shot SSL has no leash on those points beyond whatever the SSL recipe already uses.

K-means also has limits. It may not fit every data shape. The pipeline is a generic template. When you code, you may swap in a clustering method that matches the data. You are free to change that piece. The idea "cluster same-class gradient arrows" stays.

Clustering is a plug-in, not a religion.

12.10 Few-Shot VFL and Cross-Client Representation Transform

One-shot never asked the server to imagine client B's view of a person B never met. Few-shot does, because a client with only its own view cannot fill the missing view well enough to trust a label.

Only the server, which already holds overlap pairs from both A and B.

12.10.1 Why the Server Must Help

After local SSL, the client has an updated extractor. In few-shot mode it passes each unaligned row through that extractor and gets a new representation $h_A^{(u)}$. It sends that vector to the server.

Why ask the server now, when one-shot handled unaligned rows locally? Because client A only has part of person $u$. Client B has no row at all for that person, so B cannot send a matching representation. A client alone may not have enough information to generate a trustworthy pseudo-label. The paper's line, spoken in the room, is that a client with only its own view cannot fill the missing view.

The classroom picture is a split photograph: each client holds one half of each image. If the task is "are the two shapes the same?", neither half is enough to name the class. The server, which saw both halves on overlap people, can estimate the missing half.

The server does hold overlap representations from both A and B. It can study how A's overlap vectors relate to B's overlap vectors, then apply that relation to A's unaligned vector.

12.10.2 Transform T from Overlapping Pairs

Write $H_A^{(O)}$ for overlap representations from A and $H_B^{(O)}$ for overlap representations from B. Let $h_A^{(u)}$ be the unaligned representation from A. The server applies a transformation $T$ and estimates B's missing representation:

$${\hat{h}}_B^{(u)}=T(h_A^{(u)},H_A^{(O)},H_B^{(O)})$$

The hat on ${\hat{h}}_B^{(u)}$ marks an estimate, not a real vector computed on client B. Verbal description: the server estimates client B's representation for this unaligned sample in A by a transformation that uses the unaligned sample from A and all overlapped samples present in A and B.

How $T$ should think: overlap pairs $(h_A^{(o)},h_B^{(o)})$ reveal a relation. If A-side values and B-side values move together, you can read that pattern and plug in a B-side value for a new A-side point. The classroom logic is nearest overlap pattern plus the A-to-B map.

The closed-form $T$ in Sun et al. is scaled dot-product attention (a weighted sum of B's overlap rows, with weights from how close $h_A^{(u)}$ is to each overlap $h_A^{(o)}$):

$${\hat{H}}_u^B=\mathrm{softmax}\left(\frac{H_u^A(H_o^A{)}^{\top }}{\sqrt{d}}\right)H_o^B$$

where $d$ is the representation width, $H_u^A$ stacks unaligned A-side vectors, and $H_o^A,H_o^B$ are the overlap matrices. Each missing B vector is a weighted average of real B overlap vectors. Nearby A-side overlap people get large softmax weights. That is the same "who am I close to?" logic as the chalkboard map, written for vectors instead of scalars.

They pick attention rather than a generative net because overlap may be too small to train a generator, and $K$ clients would need $K$ generators.

12.10.3 Worked Mapping Example: Two Maps to Four

A toy numeric story was used again and again. It is the scalar cartoon of $T$.

Another phrasing: unaligned $h_A^{(u)}$ asks "which overlap $h_A^{(o)}$ am I close to?" Then copy that pair's relationship onto B, with a small adjustment.

12.10.4 Combining Actual A with Estimated B

Once ${\hat{h}}_B^{(u)}$ exists, the server can combine the real A vector with the estimated B vector. That pair is now a full-fledged row in representation space, even though B never stored person $u$.

The session said "combine these two". Concatenation is the natural reading, and it is the paper's $\circ$:

$$h_{\text{joint}}^{(u)}=[h_A^{(u)};{\hat{h}}_B^{(u)}]=h_A^{(u)}\circ {\hat{h}}_B^{(u)}$$

Aligned overlap rows already have both real views, so they need no $T$. Unaligned rows become usable because of $T$. Those completed rows can then enter the server classifier, subject to the trust gate in the next section.

This is how a bank-only customer might still join a joint risk model. The retail view is estimated from how bank and retail overlap customers tend to move together, not by emailing the missing receipts.

12.11 Dual-Classifier Agreement for Trusted Pseudo-Labels

Few-shot just invented a retail vector for a bank-only person. That invention can be wrong. Before the server writes a class name on $U_0$, it asks two heads and keeps the row only if they meet.

No. Missing data is better than a confident wrong label that would drift the model.

12.11.1 A-Only Classifier Versus Global Classifier

Few-shot does not blindly trust ${\hat{h}}_B^{(u)}$. The estimate can be wrong and can cause drift. The server asks two questions.

Question 1 — A-only. Pass the unaligned representation from A into a classifier that learned only A-side features. Read the predicted class and its probability. In the demo, this A-only classifier said class $1$ with probability $0.85$. Verbal description: if I pass the representation only from A, 85 percent is class one.

$${\hat{c}}_A=\arg \underset{c}{\max }{p}_A(c\mid h_A^{(u)}),p_A^{\star }=\underset{c}{\max }{p}_A(c\mid h_A^{(u)})$$

with the spoken numbers ${\hat{c}}_A=1$ and $p_A^{\star }=0.85$. Paper names: ${\hat{y}}_{u,i}^A$ and $p_{u,i}^A$ from an auxiliary head $f_c^A$ trained on overlap A-side representations.

Question 2 — global, A plus estimated B. Concatenate $h_A^{(u)}$ with ${\hat{h}}_B^{(u)}$ and pass the pair into a global classifier that learned both views. In the demo this head said class $1$ with probability $0.94$ (also spoken as $0.95$). Use $0.94$ as the written figure; $0.95$ is the same classroom point: both heads are high and close.

$${\hat{c}}_{AB}=\arg \underset{c}{\max }{p}_{AB}(c\mid h_A^{(u)},{\hat{h}}_B^{(u)}),p_{AB}^{\star }=\underset{c}{\max }{p}_{AB}(c\mid h_A^{(u)},{\hat{h}}_B^{(u)})$$

with spoken values ${\hat{c}}_{AB}=1$ and $p_{AB}^{\star }\in \{0.94,0.95\}$. Paper names: ${\hat{y}}_{u,i}^{A,B}$ and $p_{u,i}^{A,B}$ from $f_c(h_A\circ {\hat{h}}_B)$.

The paper rule taught here: the prediction is the class with maximum probability on that head. Then you compare the two heads.

Standard form (Sun et al., Eq. 9) turns the comparison into a keep-probability:

$${\hat{p}}_u^A=1({\hat{c}}_A={\hat{c}}_{AB})1(p_A^{\star }>t)1(p_{AB}^{\star }>t)p_{AB}^{\star }$$

where $t$ is a confidence floor and $1(\cdot )$ is $1$ when the statement is true, else $0$. If the heads disagree, ${\hat{p}}_u^A=0$ and the row is not sampled into the expanded labeled set. The classroom gate is the same idea in words: agree and both high, keep; disagree, reject.

12.11.2 Worked Confidence Example: 0.85 Versus 0.94

Start with four overlap samples. After a trusted unaligned fill-in, a fifth sample can join.

12.11.3 Rejecting Disagreement

Now the disagree path.

A-only head says class $1$. Global head, after adding estimated B, says class $0$ with highest probability. The two heads disagree. This sample is not trustworthy for pseudo-labeling. You do not add it to the overlap-style labeled set. You reject it.

Verbal description: if they disagree, this sample is not as trustworthy for pseudo labeling; you reject that sample; the work will compute only high quality pseudo labels for unaligned samples rather than blindly labeling them.

The gate in words:

1. Run A-only and global heads.
2. If the chosen classes match and both confidences are high, keep ${\tilde{y}}_u={\hat{c}}_A$ (which equals ${\hat{c}}_{AB}$) and add the row to the supervised set.
3. If the chosen classes differ, drop the row. Missing data is better than a confident wrong label that would drift the model.

You are not labeling by local features alone, and not by the global head alone. You ask whether they meet.

12.11.4 Student Questions and Answers

The first worry is that any predicted unaligned label can still poison training.

Growing the set is the aim. Quality is the filter.

A late recap question mixed the two table cuts.

12.12 Proto-EVFL: Prototypes, Mixed Priors, and Two-Way Losses

Sun et al. attacked limited overlap with gradients and SSL. Proto-EVFL attacks the same hole with class centres and extra care for class imbalance: one clinic may be 70% dog while the world is not.

Because a local frequency belief, used alone, lets majority classes dominate. Proto-EVFL mixes that local belief with a global one so rare classes still get samples.

12.12.1 Class Prototypes as Average Cats and Dogs

Proto-EVFL starts a different attack on the same hole: unaligned samples and missing views, now with extra care for class imbalance.

A prototype is the representative centre of a class inside one client. For the cat class on this client, take the average of how cats look here. For dog, take the average dog. For elephant, take the average elephant. Do the same on the other client, which may see a different mix.

$${\mu }_{m,z}=\frac{1}{|S_{m,z}|}\sum _{x\in S_{m,z}}{f}_m(x)$$

where $m$ indexes the client (party), $z$ indexes the class (cat, dog, elephant, ...), $S_{m,z}$ is the set of that class on that client, and $f_m(x)$ is the feature vector of row $x$. Verbal description: for this class, for this client, take the average cat — how a cat looks like on average. That is called a prototype: the representative center of a particular class.

To place an unaligned sample, ask: which class prototype is this sample closest to? Compare the unaligned vector to the cat centre, the dog centre, and the elephant centre, and pick the nearest. That is only a first match. Priors stop every leftover row from collapsing onto the biggest class.

Think of three club mascots. The cat mascot is the average of this club's cats, not a universal cat. A stray animal is compared to the mascots. The analogy breaks when the stray is a species the club never enrolled: nearest-mascot then lies, which is why priors and two-way losses appear next.

12.12.2 Local Prior, Global Prior, and Mixed Prior

A local prior is what this client believes about class frequencies. In the demo, one client was about $70\%$ dog, $20\%$ another class, and $10\%$ a third. Majority is dog inside that party. The session did not lock which animal is $20\%$ versus $10\%$ besides majority dog. Call them class $z_2$ and class $z_3$. Verbal description: what does my client think the class distribution looks like?

$$p_m(\text{dog})=0.70,p_m(z_2)=0.20,p_m(z_3)=0.10$$

A global prior is what the active party / server believes after combining all clients. Overall, cats, dogs, and elephants might be $40\%$, $35\%$, and $25\%$. Your local $70\%$ dog looks less extreme once everyone is pooled. Matching the demo numbers to "leading class versus the rest":

$$p^g(z_1)=0.40,p^g(z_2)=0.35,p^g(z_3)=0.25$$

A mixed prior is a controlled combination of local and global. You give some weight to the local view and some weight to the global view. The spoken mixed value for the leading class was $55$, which matches the average of local $70$ and global $40$:

Why mix? We do not want majority classes to dominate. If you trust only the local $70\%$ dog prior, every unaligned row tries to become a dog. Rare classes never receive matches. The mix keeps a slot open for smaller classes. Local prior plus global prior, through the mixed prior, is how rare classes still get samples.

Do not freeze on the word "prior". It is just a class-frequency belief: inside one client, across all clients, or a blend.

12.12.3 Feature-to-Prototype and Prototype-to-Feature Losses

The local extractor must improve. Two losses are combined. One spoken name was PDTC loss (also heard as PTTC). Later notes call this two-way transport cost. The published name is probabilistic dual transport cost.

The two directions are:

1. Feature to prototype: each feature vector should sit near the prototype of its class. "Which class fits this sample?"
2. Prototype to feature: each prototype should sit near the features of its class. The centre must not float away from its people.

$${\mathcal{L}}_{\text{PDTC}}={\mathcal{L}}_{f\to \mu }+{\mathcal{L}}_{\mu \to f}$$

Verbal description: compute a loss called PDTC loss — feature to prototype loss, prototype to feature — these losses are combined, then update the parameters.

Why both ways? Nearest-prototype assignment alone is greedy. Rare prototypes can be skipped so that every unaligned point attaches to dog. The reverse term says each prototype must claim some nearby points. Together they learn better features with less bias and better support for rare classes. Parameters of the local extractor move to reduce this combined loss. A small weight-decay term on ${\theta }_{E^m}$ may sit beside it in the paper; the teaching picture is the two arrows.

12.12.4 Ensemble Classifiers and Rare Classes

Input: multiple parties, different features, few aligned labeled samples, and many unaligned samples.

Flow taught at high level:

1. Extract features on each party.
2. Map each unlabeled sample to the best prototype (nearest class centre).
3. Balance class assignment using mixed priors so you cannot dump every unaligned row on one class.
4. Train party classifiers $A$, $B$, $C$ and combine them as an ensemble. Give more weight to useful representations.
5. The ensemble classifier outputs the class, including for rare classes.

Prototypes are not the final output. They are intermittent guidance to improve learning. The ultimate goal is the classifier that names the class of an unaligned sample. The paper adds an adaptive gate when combining parties so one party's features cannot drown another party's rare-class signal. The exam-level picture is: prototypes guide; the ensemble names.

12.12.5 What to Take Away Before the Next Session

Time ran out for a full Proto-EVFL derivation. The next session returns to it in more depth, including the unaligned versus aligned split and local priors.

What to keep now:

• Horizontal = same features, split rows.
• Vertical = split columns, limited overlapping people, missing views.
• The Duke-NVIDIA method: representations up, labeled loss on the server, gradients down, k-means pseudo-labels, local SSL; one-shot enriches, few-shot estimates the missing view and keeps a label only if two heads agree.
• Proto-EVFL: average class centres, mix local and global class rates, pull features and centres toward each other, then classify.

Exam Guidance Summary

Marked presentation materials for these algorithms are expected to be available during the exam. You are not asked to reproduce every variable.

Score comes from high-level flow. What is overlapping versus unaligned? Why mean-fill fails on images? What travels (representations, gradients, class count $C$) and what never travels (raw data, true labels)?

Know the crux phrase same-class gradients point the same way, and that k-means with $K=C$ builds overlap pseudo-labels.

Contrast one-shot VFL (unaligned rows enrich representations, no explicit unaligned label) with few-shot VFL (estimate missing view with $T$, dual-classifier agreement, reject on disagreement). High-level understanding of flow, advantages, and drawbacks is what is scored.

For Proto-EVFL, know prototype = class average, local prior versus global prior, mixed prior so majorities do not dominate, and two-way feature-prototype loss. Tough questions from the unfinished Proto-EVFL pass are not the plan for this sitting. Analysis of efficiency versus drawback is.

Do not memorize theta sequences. Do analyze which method uses almost all data, and which checks keep pseudo-labels high quality.

Key Industry Applications

Banks, retail stores, and other firms hold different columns on partly overlapping customers. Joint risk labels need both finance features and purchase features. That is the running vertical demo in this session.

Heart, eye, and skin clinics (or medical-image sites) each hold one view needed for a health index. Raw scans should not leave the clinic. Mean-fill cannot replace a missing scan.

A client office in one city (spoken example: Hyderabad) and a server in another country (spoken example: the US) may both refuse a full data share. Vertical federated learning is the pattern that still trains a joint model.

The communication-efficient limited-overlap VFL paper comes from Duke University authors with NVIDIA. NVIDIA appears here as an industry co-author, tying the classroom method to production-scale federated research (including an NVFlare one-shot VFL example).

Phone keyboards remain the stock horizontal industry picture: same typing features, different users, model averages rather than raw keystrokes.

Vertical federated learning has been applied in company settings well enough to earn internal awards and later an appreciation mail from a student who shipped it at work.

Companion textbooks also point to bank plus e-commerce credit models and to medical diagnosis settings where sites cannot centralize records. The same geometry applies: different attributes, shared people where IDs match, labels often on one active party.