Model Caching for Decentralized Federated Learning

3.1 Recap: Parallelism, Batches, and Caching

3.1.1 Parallelism and the need for a cache

Splitting work across many computing units is called parallelism (doing many parts of one job at the same time on separate workers). The lecture named three flavours. Data parallelism (each worker gets a different slice of the data) hands worker A rows 1 to 1000 and worker B rows 1001 to 2000, while both hold a full copy of the model. Model parallelism (each worker holds a different part of one model) places layers 1 to 5 on worker A and layers 6 to 10 on worker B, while the same data flows through both. Code parallelism (different program steps run on different workers) spreads the pipeline stages themselves. All three buy speed in theory and then meet the same bottleneck in practice: a worker that stalls on slow storage throws away the speed that parallelism bought.

A cache (a small, fast store kept close to the compute) removes that wait. Think of a cook who keeps salt, oil, and a sharp knife on the counter instead of walking to the pantry for every dish. The counter is small but instant; the pantry is large but slow. Training hardware has the same two levels. The in-memory cache lives in RAM and answers in nanoseconds. The on-disk cache is larger but slower by orders of magnitude. The strategy is to load the next data chunk into the fast cache before it is needed, so the model running in RAM finds its input already waiting. When the access pattern is known ahead of time, this preloading turns a storage-bound job into a compute-bound one.

In plain words: parallelism decides who does what, while the cache decides where the next input waits. One without the other leaves speed on the table. The batch order of epoch training is what makes the cache plan possible, which is the next point.

3.1.2 Known order makes prefetch work

Training in epochs (full passes over the data) gives exactly that known pattern. An epoch is one full sweep: if a dataset has 1000 samples and each batch (a small group of samples processed together) holds 250 samples, then one epoch holds 4 batches. The order of batches is fixed before training starts. Batch 2 is processed after batch 1, batch 3 after batch 2, and so on. Because the batch order is known, the system can hold the current batch in the in-memory cache while fetching the next one. Each batch is placed into RAM, the model consumes it, and the response stays fast.

Picture the access timeline with batch order on the horizontal axis (batch index 1, 2, 3, 4) and time on the vertical axis. Without prefetch the shape is a staircase with flat gaps: compute, pause for load, compute, pause for load. With prefetch the shape is a smooth ramp: the load bar for batch $k+1$ overlaps the compute bar for batch $k$. The landmark to watch is the overlap region — wherever load hides under compute, the gap is gone. The takeaway in one sentence: known batch order lets load and compute overlap, so the cache hides storage delay.

If the order is random with no lookahead, or the cache holds only one batch, the overlap breaks and stalls return. Video streaming buffers and database buffer pools use the same idea for the same reason — know what comes next, and fetch it early. That known-order trick is the bridge to the main story: in decentralized learning the "next item" is not the next batch but the next peer model, and a model cache plays the same hiding role.

3.2 Decentralized Learning on Mobile Agents

3.2.1 Agents, devices, and local datasets

A mobile agent (any compute node that moves, senses, and learns where it stands) can be a phone, a sensor, an edge box, a small board such as a Raspberry Pi, a drone, a vehicle, or a factory machine. Call the devices $m_1,m_2,\dots ,m_n$, where each $m_i$ is one agent and $n$ is the total count. In set form:

$$\mathcal{M}=\{m_1,m_2,\dots ,m_n\}$$

Here $\mathcal{M}$ is the fleet (the full set of agents), $m_i$ names agent number $i$, and $n$ counts the agents. Every agent owns a private local dataset that keeps growing as the device senses the world. One device collects rows of heart-related readings, another collects skin-related readings, a third collects readings from other instruments. Each row pairs input features with their labels. A heart row might pair features $x=$ [age, resting pulse, blood pressure] with label $z=$ healthy or at-risk. A skin row pairs a different feature list with its own label. No two agents promise the same columns.

Decentralized means there is no central machine that sees all the data. The devices are scattered, they move, and any two of them can exchange information only while they are physically close enough to communicate — inside each other's communication range. Outside that range, each device is on its own. Picture a map with dots for agents and short circles for radio reach: an edge appears only where two circles overlap. The graph of who can talk to whom changes every minute as agents move.

This setup matches the standard federated picture of agents with local datasets and a shared model goal, except there is no always-on server. Phones that learn typing habits on-device and factory machines that learn vibration patterns locally both follow this shape: sense where you stand, learn where you stand, share only what you learned.

3.2.2 The combined-model goal

Each agent trains its own model on its own data — a small neural network, a convolutional network, or any model family suited to the task. The dream is a combined model: feed it all three feature groups at once — heart features, skin features, instrument features — and get the best possible accuracy at test time. Yet the combined data is never assembled in one place. No device ships its raw rows anywhere. Devices share model updates only: weight vectors learned locally. The combined model must be built out of shared weights, never out of pooled data.

An everyday picture helps. Think of three cooks who each know one cuisine and never share their secret recipes, but mail each other tasting notes. Each cook adjusts their own menu from the notes and slowly all three menus cover all three cuisines. The mapping is direct: secret recipe = private dataset $D_i$, tasting notes = shared weight vector, menu = local model. The picture breaks in one place: tasting notes are vague, while weight vectors carry precise numeric updates that training can actually use.

Hospital instruments that learn from scans without exporting patient rows show why the goal matters: privacy law forbids pooling, but patients would still gain if heart, skin, and instrument knowledge lived in one model. That is the agents dataset combined model problem in one line — many private datasets, one shared model, no raw data movement.

3.2.3 Student questions on centralization and orchestration

The confusion is natural because the word central appears in both ideas. Centralized training means data moves to one machine and gradients are computed there. Here data never moves; only weight vectors move, and the heavy work of fitting data stays on the agents. An orchestrator that averages vectors is closer to a post office sorting letters than to a kitchen cooking the meal.

3.3 Why Model Caching: Slow Devices and Asynchronous Training

3.3.1 When updates arrive late

Real deployments break the tidy assumption that every device answers on time. Three facts dominate. Devices are slow — a sensor or an old phone trains at its own pace, perhaps one local epoch per hour while a plugged-in edge box does ten. Communication is unstable — a device may send its update now and go silent for hours as it drives through a dead zone. Updates arrive delayed, and any scheme that waits for the newest model from every peer stalls. Perfect synchronization, where each client always holds the latest copy of every other client's model, is not possible across scattered, moving hardware.

Asynchronous training (each device keeps learning whenever it can, with whatever peer information it currently holds) accepts this reality. Synchronous training (all devices pause at a barrier until the slowest peer catches up) is the alternative that fails here: one straggler holds the whole fleet hostage. Model caching is what makes the asynchronous style work. Instead of freezing while a peer is out of reach, a device trains with a cached — slightly old — copy of that peer's model and swaps in the fresh copy when contact resumes. Those cached models, not fresh copies, are what each round actually aggregates.

3.3.2 Cached models keep training moving

The rule is simple: never wait for the newest peer model; train with the cached one for now. Suppose device C1 wants device C3's latest weights but C3 is out of range. C1 keeps training with the older C3 weights sitting in its cache. Later, C1 and C3 drift into range and exchange caches directly. Or the update arrives by relay: C3 meets C2 and hands over its cache, then C1 meets C2 and picks up C3's newer weights second-hand. Either path delivers the update, and no device ever idles waiting.

Because of this tolerance for stale information, the system scales: new agents can wander in, swap caches with whoever they meet, and join the shared effort. A fourth phone joining a street needs only one meeting to get a useful starting set of peer models, not a full fleet handshake.

One caution was stated directly and must not be lost: scalability improves, but robustness is not guaranteed — old models can still pull training off course if they are used too long. That danger is exactly what the staleness bound in a later section controls. In other words, stale models keep the engine running, but stale models past their expiry date steer it off the road.

3.3.3 What caching buys: stability, less waiting, lower overhead

Four gains were named. Cached models replace perfect synchronization, which cannot be achieved. Stale updates are tolerated instead of blocking progress. Training stabilizes because every device always has something reasonable to aggregate — the average never collapses to one lonely local model. Communication overhead falls because an unchanged model need not be resent — the peer's cached copy is already correct, so the radio stays quiet. A device whose weights barely moved between rounds simply stays silent, and peers reuse the cached copy.

Picture a chart with wall-clock time on the horizontal axis and number of idle devices on the vertical axis. The synchronous line spikes to full idle every round while waiting for stragglers. The cached async line stays near zero idle, with small bumps at cache swaps. The takeaway in one sentence: use-old-until-fresh removes the wait without removing learning.

Autonomous vehicles that pass each other briefly, delivery drones with patchy links, and remote field sensors all depend on this use-old-until-fresh discipline. Each keeps a small desk drawer of peer models (the cache analogy from the opening recap) and works from the drawer until a fresh delivery arrives.

3.4 Learning Locally While Staying Close to the Shared Model

3.4.1 The two aims side by side

Every node serves two masters at once. First, it must learn from its own local data — fit the patterns its own sensors actually see. A heart-device must nail heart rhythms; a skin-device must nail skin textures. Second, it must stay close to the global model, the decentralized consensus that averages what all peers have learned. Drift too far from the consensus and accuracy collapses, because the lone model forgets everything the other devices know. The averaged model never exactly equals the model one central machine would have trained on pooled data, but the aim is to land on par with it — close in performance, not identical in weights.

Think of it like a study group, the lecture's own picture. Each member masters their own chapter deeply, then checks their notes against the group's shared summary. A member whose notes contradict the summary on every page has probably overfit to their chapter — memorized chapter quirks instead of learning the course. The fix is to keep personal notes rich but compatible with the shared version. The mapping is exact: own chapter = local data, personal notes = local weights, shared summary = consensus model. The picture breaks where all analogies break: students can argue in words, while devices can only argue in averaged numbers.

3.4.2 Student questions and answers

Devices agree on the blueprint once, then trade only the numbers that fill it in. That is why the distance math in a later section can compare weight vectors point to point — the layouts match.

Several students circled the same doubt in different words, and the answer stayed the same: the cache is the memory that bridges meetings.

3.4.3 Privacy: data stays, models travel

Because rows never move, sensitive sources can collaborate. Hospitals keep patient scans behind their own walls yet contribute heart or skin knowledge through weights. Banks keep account rows private yet contribute fraud patterns through weights. Factories keep vibration logs on the machines yet contribute fault patterns through weights. The shared model still benefits from all of them. That privacy property is a direct consequence of the two-aim setup, not an add-on: local fit never needs to export rows, and consensus only needs to import weights.

3.5 Non-IID Data: Same Features Versus Mixed Features

3.5.1 IID and why it matters

Classical training assumes IID data — independent and identically distributed. Independent (each sample is drawn without influence from the others) means seeing one sample tells you nothing extra about the next beyond the shared pattern; there is no hidden copying. Identically distributed (every sample comes from the same probability distribution, the same data-generating process) means every batch looks statistically alike. One student first offered "unique and non-identical," which mixes the idea up; the correction is that samples must be independent of each other and identical in distribution, not unique. Under IID, a batch from Tirupati looks statistically like a batch from Bhopal, and one model fits all of it.

Here that assumption breaks. One device sees heart features, another sees skin features, a third sees eye features. The distributions differ by construction. Such data is called non-IID — not independent and identically distributed across devices — and also heterogeneous (drawn from different processes with different features). The shared model must reconcile sources that disagree about which features even exist. Independent same distribution IID samples are the textbook ideal; heart, skin, and eye rows are the opposite.

3.5.2 Homogeneous case: same schema, simple problem

When every participant records the same features, life is easy. Two branches of one bank, in two cities, log the same columns: balances, transactions, loan flags. Regulations may differ across regions, yet the schema matches, so the data is homogeneous (same columns, same meaning) and nearly IID. Merging models from such branches is straightforward, and even simple averaging performs well. This is the easy regime, and it is not the target of the caching design.

A marble-bag picture helps. Homogeneous means every bag holds red and blue marbles in about the same mix; a handful from any bag predicts the others. Heterogeneous, coming next, means one bag holds marbles, another holds dice, and a third holds cards — handfuls do not predict each other at all. The picture breaks in that real features are richer than colors, but the sameness idea carries over.

3.5.3 Heterogeneous case: mixed schemas, hard problem

The hard regime — the actual target — mixes unlike sources. A conglomerate wants one financial index from banking records plus loan-repayment records plus transaction streams, each living in a different arm of the business. In health care, one hospital group excels in heart care, another in skin care, a third in eye care, and the goal is one health index per person from all three feature groups. Nobody will pool raw rows, so each party trains on its own slice and shares models only. The shared model must then perform well on combined inputs it never saw together during training — a heart-plus-skin-plus-eye row at test time, built from devices that each saw only one slice.

Pick homogeneous tools for same-schema branches; pick the caching design for unlike-feature conglomerates.

Card, loan, and payments arms of one financial group building a joint risk index, heart, skin, and eye specialty hospitals building a joint health index, and insurance, banking, and manufacturing arms each contributing their own columns all show the same hard shape.

3.6 Weighted Aggregation with Neighbor Importance

3.6.1 Mathematical formulation

The lecture's plain-language rule was: larger alpha for neighbor $j$ means more influence of $j$ on my updated model. The reconstruction is a weighted average over the models currently held for the neighborhood. Here $x_j$ is the weight vector of neighbor $j$ (a list of $d$ numbers, one per model parameter), the set $N(i)$ holds the neighbors device $i$ currently knows through its cache, each ${\alpha }_j$ is a non-negative importance weight for neighbor $j$, and $x_{i,t+1}$ is device $i$'s new collaborative model for the next round:

$$x_{i,t+1}=\sum _{j\in N(i)}{\alpha }_j{x}_j$$

with the normalization condition:

$$\sum _{j\in N(i)}{\alpha }_j=1,{\alpha }_j\ge 0$$

Each ${\alpha }_j$ names neighbor importance (how much device $i$ trusts neighbor $j$ this round). The sum-to-one normalization keeps the blend on the same scale as the inputs: without it, weights like 0.6, 0.3, 0.1 would still work only because they already sum to 1.0, while raw scores like 6, 3, 1 must first be divided by their total 10. In standard federated averaging the same idea appears as weighting by dataset size, ${\alpha }_j=n_j/N$ with $n_j$ samples on client $j$ and $N$ total samples, which also sums to one by construction. Simple averaging, where every neighbor counts equally with ${\alpha }_j=1/|N(i)|$, is the special case of equal trust; the weighted form generalizes it by letting trustworthy or data-rich neighbors count more. Larger alpha pulls the result toward that neighbor — that pull is the whole point of weighted aggregation with neighbor importance blend.

Compared side by side: simple averaging gives each of $K$ neighbors weight $1/K$ and needs no tuning but lets a stale peer drag the mean; weighted averaging gives data-rich or fresh peers larger $\alpha$ and stays accurate under heterogeneity but needs a policy for picking $\alpha$. When peers look alike and fresh, pick equal weights; when peers differ in data size, freshness, or trust, pick tuned importance weights.

3.6.2 Worked numbers: 0.6, 0.3, 0.1

A concrete assignment was walked through. Device $i$ holds cached models from three peers, C1, C2, C3. It assigns importance $0.6$ to C1, $0.3$ to C2, and $0.1$ to C3. The new model is then $0.6$ times C1's weights plus $0.3$ times C2's weights plus $0.1$ times C3's weights, computed coordinate by coordinate over the whole weight vector.

Picture a bar chart with neighbors C1, C2, C3 on the horizontal axis and alpha on the vertical axis from 0 to 1. The bars stand at 0.6, 0.3, 0.1 — a tall, medium, and short bar. The landmark is the 0.6 bar holding more than half the mass: wherever that bar points, the blend follows. The takeaway in one sentence: bar height is pull strength.

Choosing these weights well is itself a design problem; many aggregation algorithms exist beyond plain averaging, from data-size weighting to trust scores to staleness penalties, and picking among them is where much of the complexity lives. A common beginner trap is leaving raw scores unnormalized (using 6, 3, 1 directly inflates the model by 10 times); always divide by the total first. A second trap is freezing alphas forever while peers drift in freshness; re-pick them each round.

3.6.3 Student questions and answers

The same confusion returns in new words, so it is worth locking in: plain average is weighted average with all alphas equal.

There is no central whiteboard; there are $n$ personal whiteboards that slowly agree.

In short, alpha weights aggregation is personal: my cache, my alphas, my next model — and the fleet agrees by repeating that personal step.

3.7 Local Update in Full: Two-Part Objective and K Steps

3.7.1 Mathematical formulation

The local-update function performs training on one node $i$ at one global round $t$. Its plain-language specification was: take care of two loss functions — one that trains on the data, and one that penalizes moving away from the global model. The reconstruction uses $x$ for the model weight vector being trained (in ${\mathbb{R}}^d$), $f(x;\cdot )$ for the ordinary task loss on local samples, ${\bar{x}}_t$ for the consensus (global) weight vector at round $t$, and $\rho$ for the penalty strength, a non-negative scalar pronounced "row" in the session:

$$L_{i,t}(x)=f(x;\xi )+\frac{\rho }{2}\Vert x-{\bar{x}}_t{\Vert }^2$$

where $\xi$ is the current mini-batch of local samples. The first part fits the data. The second part, the proximal or regulation term, grows with the squared distance between the local weights and the consensus weights, so wandering far gets expensive. The one-half factor is the standard convention: since the gradient of $\frac{1}{2}\Vert x-{\bar{x}}_t{\Vert }^2$ is exactly $(x-{\bar{x}}_t)$, the half cancels the factor of 2 from differentiating a square and leaves a clean penalty gradient $\rho (x-{\bar{x}}_t)$. Reference code for this family writes the same term as $\mu \Vert w-w^t{\Vert }^2$ with $\mu$ in place of $\rho /2$; here we keep the lecture's $\rho$ with the half to match the session, which is the same shape with $\mu =\rho /2$.

The single gradient-descent step on this two-part local objective with penalty consensus, with learning rate $\eta$, reads:

$$x_{i,t}^{(k)}=x_{i,t}^{(k-1)}-\eta \nabla L_{i,t}(x_{i,t}^{(k-1)};{\xi }_{i,k})$$

in words from the session: new weight equals old weight minus the learning rate times dou L by dou W, where dou L by dou W is the gradient of the loss with respect to the weights. The learning-rate symbol $\eta$ is the standard choice for step size; the session called it a constant learning rate (heard as "n"), and $\eta$ is that same constant in standard notation. Here $\nabla L_{i,t}$ is the gradient vector (one partial derivative per weight), $x_{i,t}^{(k-1)}$ is where the step starts, ${\xi }_{i,k}$ is the mini-batch used for the step, and $\eta >0$ scales how far the step moves. Unfolding the gradient step learning rate weights update in full:

$$\begin{array}{rlr}\nabla L_{i,t}(x;{\xi }_{i,k})& =\nabla f(x;{\xi }_{i,k})+\rho (x-{\bar{x}}_t)& \text{(gradient of data part plus gradient of penalty)}\\ x_{i,t}^{(k)}& =x_{i,t}^{(k-1)}-\eta \nabla f(x_{i,t}^{(k-1)};{\xi }_{i,k})-\eta \rho (x_{i,t}^{(k-1)}-{\bar{x}}_t)& \text{(step against both pulls at once)}\end{array}$$

Think of a dog on a leash: the dog (local weights) wants to chase every interesting smell (local batches), while the owner (consensus) holds the leash (penalty) at ${\bar{x}}_t$. A short leash (large $\rho$) keeps the dog at heel; a long leash (small $\rho$) lets it roam. The leash never breaks — it only stretches at a price.

3.7.2 Classification loss or regression loss

The task loss $f$ is chosen by problem type. For classification — for example, naming the object in a phone photo — use cross-entropy loss between predicted class scores and true labels. With true label $z$ (ground truth, say $z=1$ for cat) and predicted probability $\hat{z}$ (say $\hat{z}=0.7$ for cat), the loss is $-\log \hat{z}\approx 0.36$; a confident wrong answer like $\hat{z}=0.1$ pays $-\log 0.1\approx 2.30$. For regression — for example, predicting a sensor reading — use mean-squared-error loss between predicted and true values: with truth $z=20.0$ and prediction $\hat{z}=18.0$, the loss is $(18.0-20.0{)}^2=4.0$. Here $x$ names a single input sample's features, $z$ names its true label, the model maps $x$ to a prediction often written $\hat{z}$, and the loss measures the gap between $\hat{z}$ and $z$. Any other task-appropriate loss fits the same slot.

3.7.3 The penalty term as a clamp

Without the penalty, non-IID data plus delayed peer updates lets local weights sprint off alone. The penalty clamps that sprint — it acts like a speed limiter that charges more the farther weights stray.

The strength $\rho$ tunes the grip. A large $\rho$ clamps hard and keeps the model glued to consensus; a small $\rho$ loosens control and lets local patterns dominate. Picture a plot with weight value on the horizontal axis and total loss on the vertical axis: the data loss is a valley near 100, the penalty is a valley near 10, and the sum is a valley somewhere between, sliding toward 10 as $\rho$ grows. Under heterogeneous data and stale peer information, the clamp is what keeps the shared effort from flying apart.

A beginner trap is setting $\rho =0$ "to learn faster" on skewed data — the model then memorizes its own slice and fails the joint test. The opposite trap is cranking $\rho$ sky-high so the model never leaves consensus and learns nothing local. Tune in the middle and watch joint accuracy, not just local loss.

3.7.4 Student questions and answers

Regularization here does not mean weight decay toward zero; it means decay toward the shared model.

Phone keyboards that adapt to one user's slang without forgetting standard language, factory monitors tuned to one machine that must still agree with the fleet-wide fault model, and hospital models tuned to local patients that must stay compatible with the joint index all live on this same leash.

3.8 Symbols, Batches, and the Update Rule Step by Step

3.8.1 What i, t, and k name

Three indices remove all ambiguity, and they are worth memorizing. The subscript $i$ names the device: node $i$, client $i$, agent $i$ are the same thing. The subscript $t$ names the global round, also called the global epoch: the $t$-th outer cycle of the whole system. The superscript $(k)$ counts local steps inside that round: the $k$-th mini-batch update on device $i$ during round $t$. So $x_{i,t}$ is the model where device $i$ starts round $t$, $x_{i,t}^{(k)}$ is the model after $k$ local updates, and $D_i$ is device $i$'s private local dataset, which is never shared. Local step indices with device round batch addressing mean no two updates share a name: $x_{2,7}^{(3)}$ can only mean device 2, round 7, after 3 local steps.

3.8.2 Sampling a mini-batch

Training consumes the dataset $K$ mini-batches at a time. At local step $k$, the device draws one mini-batch at random from $D_i$; call it ${\xi }_{i,k}$, a small bundle of samples, each pairing features $x$ with a label $z$. Phone photos split into cat-image and dog-image batches, factory logs split into sensor-signal batches, and hospital archives split into scan batches are the running examples. Because fresh rows keep arriving on the device itself, sampling always reflects the latest local reality. Mini-batch sampling keeps memory small and gradients noisy in a helpful way: small bundles mean many updates per round instead of one giant slow one.

3.8.3 Gradient step and repeat-K loop

One local step runs the same five moves. Start from the current weights. Feed the mini-batch through the model to get predictions. Score predictions against true labels with the task loss and add the drift penalty. Differentiate that combined loss with respect to every weight to get the gradient vector. Subtract the learning-rate-scaled gradient from the old weights to get the new weights. Repeat for $k=1,2,\dots ,K$, then hand back the final weights as the device's trained model for this round.

The cost is easy to state: $K$ gradient evaluations per round per device, each over one mini-batch, so double $K$ roughly doubles local compute. The practical limit is drift: large $K$ lets a lonely device wander far before the next aggregation, which is why the penalty term exists.

3.8.4 Worked walkthrough of one step

This is ordinary mini-batch training, identical in mechanics to single-machine learning — the distributed novelty lies entirely in the penalty term and in where batches come from. When to use this loop as-is: small $K$ (say 5 to 20) with fresh consensus nearby. The alternative of huge $K$ with no penalty belongs only to IID, always-connected setups, which this lecture explicitly leaves behind.

3.9 Cache Exchange, Staleness Bound, and Freshness

3.9.1 What each cache holds

Every device keeps a small memory holding its own latest model plus the latest models it has received from others. An entry is more than bare weights: it carries which agent the weights came from and a timestamp or version marking when they were produced. A cache might read: my own model at version 5, C2's model at version 4, C3's model at version 1, C4's model at version 3. Because devices meet only pairwise and sporadically, two devices' caches usually disagree about some third device — and that disagreement is normal, not a bug. Think of travelers swapping notebooks: each notebook lists who wrote each page and when, and no two travelers hold identical sets for long.

3.9.2 Relay through a middle device

The relay story was told with clocks. C1's cache holds C3's model stamped 1 o'clock. C3 later meets C2 and shares its fresher 2 o'clock model. Now C2's cache holds C3-at-2 while C1 still holds C3-at-1. When C1 and C2 meet, they swap full caches. C1 compares the two C3 entries, sees 2 o'clock beats 1 o'clock, drops the older entry, and keeps the newer one. Symmetrically, C2 absorbs anything fresher from C1's cache. Through such pairwise swaps, fresh models diffuse across the fleet without any device ever contacting the whole fleet at once. The relay with fresher cache diffuse fleet behavior means news hops person to person: C3 tells C2, C2 tells C1, and C1 ends up fresh without ever meeting C3.

3.9.3 Staleness arithmetic and the cutoff

The session's plain words were: t minus tau is the time delay — how old the model is. A neighbor's update came from round 3, the current round is 7, so staleness is 4. With $t$ for the current round and ${\tau }_j$ for the round stamped on neighbor $j$'s cached model, staleness is:

$$s_j=t-{\tau }_j$$

Here $s_j$ is the staleness (age in rounds) of neighbor $j$'s cached model, $t$ is now, and ${\tau }_j$ is when that model was born. A cached model survives only while it is fresh enough:

$$s_j\le {\tau }_{\max }$$

where ${\tau }_{\max }$ is the maximum tolerated delay. The symbol ${\tau }_{\max }$ is the standard name for this cutoff: the session described setting "this value" as the cutoff age, and ${\tau }_{\max }$ is that same value written in symbols. Work it through: current round $t=7$, cached stamp ${\tau }_j=3$, so $s_j=7-3=4$. If the cutoff ${\tau }_{\max }=5$, the model stays since $4\le 5$; at staleness $6$ or more it is discarded. A clock-time version said the same thing: keep a received model for five hours, then remove it, because training on older weights risks huge drift. Five hours remove model huge drift is the wall-clock twin of the round-count rule. The cutoff is the safety valve that answers the robustness worry from the earlier section: staleness current round cached stamp cutoff arithmetic decides, and fresh cutoff delay model survives logic enforces it.

Picture staleness on the horizontal axis (0 to 8 rounds) and usability on the vertical axis (keep or drop). The curve is flat at keep until ${\tau }_{\max }=5$, then drops like a cliff to drop. The landmark is the cliff edge at 5: a model aged 4 sits safely left of the edge, a model aged 6 lies off the cliff. The takeaway in one sentence: age in, cliff decides.

Set the cutoff too large and ancient models steer training off course; set it to zero and the cache empties between meetings so devices train alone. The sweet spot keeps the freshest affordable set.

3.9.4 Cache update procedure

The update runs as a fixed procedure on every meeting:

1. Collect each newly received model with its sender id and timestamp.
2. For each cached entry, check its age $t-{\tau }_j$ against the cutoff and delete entries older than the maximum.
3. For each received model, look for an existing entry from the same agent. If none exists, store it.
4. If an entry exists, compare timestamps and keep whichever is latest, replacing the older one.
5. Sort the surviving entries by recency and keep only the first $C$, the cache capacity, returning the trimmed cache.

This runs before aggregation, so aggregation always blends the freshest affordable set of peer models. Complexity is tiny — a handful of timestamp compares per meeting — and the cost it controls is radio and memory: small $C$ and tight ${\tau }_{\max }$ mean less storage and less risk, at the price of less diversity. Use this procedure whenever meetings are pairwise and random; the alternative of waiting for a full-fleet sync belongs to wired clusters, not moving agents. Cache staleness cutoff policing is what makes that choice safe.

That exam skill is the operational half; the conceptual half is what the next section measures.

3.10 Measuring Drift with Euclidean Distance and the Shared Training Goal

3.10.1 Distance between two weight vectors

The plain-language statement was: we compute the distance between the two models — current model versus cached model — using Euclidean distance. With $x$ for the current weight vector and $x_{\text{cache}}$ for the cached peer vector, both in ${\mathbb{R}}^d$ (lists of $d$ real numbers with the same layout):

$$d=\Vert x-x_{\text{cache}}{\Vert }_2$$

Here $d\ge 0$ is the drift distance, $x$ is the local model, $x_{\text{cache}}$ is the peer model from the cache, and $\Vert \cdot {\Vert }_2$ is the Euclidean norm (square root of summed squares). The norm expands coordinate-wise over Euclidean distance weight vectors models entries. With $x=[x_1,\dots ,x_d]$ and $x_{\text{cache}}=[c_1,\dots ,c_d]$:

$$d=\sqrt{\sum _{k=1}^d(x_k-c_k{)}^2}$$

Each $(x_k-c_k)$ is the per-weight gap, squaring removes sign, adding pools all gaps, and the square root returns to the original units. A small $d$ means the models agree; a large $d$ means the local model has wandered. Minimizing such gaps is precisely what the penalty term in the local objective does at training time — the penalty is the squared version of this same distance.

Picture distance on the horizontal axis from 0 outward and loss penalty on the vertical axis. The curve is a parabola through the origin: flat near zero drift, steep far out. The landmark is the elbow where small gaps cost little and large gaps cost a lot — that elbow is where the clamp starts to bite hard.

3.10.2 Why equal architecture is required

A size mismatch breaks the math before it starts: a 100-weight vector minus a 120-weight vector has 20 gaps with no partner. That is why architecture is fixed once and only weights travel.

3.10.3 Global training objective

Zooming out, the whole fleet optimizes one shared goal. In the session's words: find the model weights that minimize the loss — take any neighbor's dataset, take samples, feed them to the model, and the loss must be small. With $x\in {\mathbb{R}}^d$ for the $d$-dimensional real weight vector, $D_i$ for neighbor $i$'s dataset, $\xi$ for a sample drawn from it, and $\ell$ for the per-sample loss such as cross-entropy:

$$\underset{x\in {\mathbb{R}}^d}{\min }{\mathbb{E}}_{i,\xi \sim D_i}[\ell (x;\xi )]$$

Read it inside out: pick a participating neighbor $i$, draw a sample $\xi$ from its private data $D_i$, run the model $x$ on it, score the loss $\ell (x;\xi )$, and average over all such draws. The best weights make that global objective expectation average loss as small as possible. Every device's local penalty-pulled training plus weighted aggregation is a decentralized attempt to approach this single minimum without ever centralizing the data. In numbers: three neighbors with mean losses 0.2, 0.5, 0.9 give overall mean $(0.2+0.5+0.9)/3\approx 0.53$; training chases weights that push that 0.53 down.

A beginner trap is reading the expectation as one fixed batch average — it is the mean over the whole fleet process, not one device's round. Another trap is chasing local loss to zero while the global mean rises; the penalty and aggregation exist to stop exactly that.

3.10.4 What capital E names

The slot-machine picture from probability helps: expectation is the long-run average payout, not one lucky pull. Here the payout is negative loss, and training picks the machine settings with the best average.

3.11 End-to-End Loop: Local Training, Cache Update, Aggregation

3.11.1 The three functions in order

The complete method is three functions repeating forever. First, local update: each client trains its own data through $K$ penalized gradient steps. Second, cache update: each client gathers neighboring and cached models that are not too old, dropping whatever exceeds the staleness cutoff. Third, model aggregation: each client blends the surviving neighbor models with its importance weights into an improved collaborative model for the next round. All clients run all rounds until final trained models emerge on every device. Those three functions repeating cycle form the heartbeat: train, refresh memory, blend — then again.

3.11.2 A round from start to finish

Follow device $i$ through one full cycle. It starts round $t$ from its current model. It samples $K$ mini-batches from $D_i$, stepping weights forward each time while the penalty term tugs toward consensus. It then meets whoever is in range, swaps caches, discards over-age entries, and keeps the freshest $C$ models. It forms $x_{i,t+1}$ as the $\alpha$-weighted blend of those survivors. That blend becomes the starting point of round $t+1$, and the cycle repeats. No step ever waits for the full fleet, which is why the loop survives slow hardware and fickle links.

Walk the timeline once with small numbers: round $t=7$ starts at $x_{i,7}$; $K=3$ local steps move it with batches ${\xi }_{i,1},{\xi }_{i,2},{\xi }_{i,3}$; a meeting swaps caches and drops a C3-at-1 entry aged $7-1=6>5$; survivors C2-at-6 and C4-at-5 blend with $\alpha =[0.6,0.4]$ into $x_{i,8}$. One line per job, one job per meeting, no barrier anywhere.

Picture rounds on the horizontal axis and model quality on the vertical axis. Each local-update segment climbs a little, each cache-plus-aggregation segment jumps toward the fleet mean, and the sawtooth trends upward. The landmark is the jump at each blend: without it the lines wander apart; with it they ratchet together.

3.11.3 Open thread: slow convergence

Slow here never means desirable — it names the cost of random meetings. Caching does not remove the randomness; it softens its price by always giving each device something to blend.

Exam Guidance Summary

No mark split, question pattern, or exempted topic was stated in this session, so there is no distribution to report. The guidance below collects the emphasis signals actually given:

• State the twin aims — learn from local data, stay close to the consensus — and explain the accuracy cost of dropping either one. Dropping local fit wastes fresh on-device data; dropping consensus lets drift wreck shared accuracy. This is the central exam focus.
• Define IID (independent draws from the same distribution) and contrast homogeneous same-schema collaboration (two bank branches, same columns, easy averaging) with heterogeneous mixed-schema collaboration (banking plus loan plus transaction arms, or heart plus skin plus eye hospitals, hard joint index), each with one example.
• Write the two-part local objective in words and symbols, $L_{i,t}(x)=f(x;\xi )+(\rho /2)\Vert x-{\bar{x}}_t{\Vert }^2$, name every symbol ($x$ weights, $f$ task loss, $\xi$ batch, ${\bar{x}}_t$ consensus, $\rho$ grip), and explain how the penalty strength $\rho$ tightens or loosens the clamp.
• Perform one weighted-aggregation computation from stated importance weights (check alphas sum to one, multiply coordinate-wise, add) and interpret what a larger $\alpha$ does — it pulls the blend toward that neighbor.
• Compute staleness as current round minus stamped round ($s_j=t-{\tau }_j$), apply a cutoff (${\tau }_{\max }$), and justify discarding old models through drift: stale weights pull the blend toward outdated patterns.
• State the Euclidean drift distance ($d=\Vert x-x_{\text{cache}}{\Vert }_2$), its same-architecture precondition (same $d$, same layout, or subtraction is undefined), and the shared global objective with expectation (${\min }_x{\mathbb{E}}_{i,\xi \sim D_i}[\ell (x;\xi )]$, the mean loss over neighbors and samples).
• Carry the open convergence thread: inconsistent sharing slows agreement because models move and meetings are random, and caching with fresh cutoffs is the proposed remedy.

Key Industry Applications

• Autonomous driving stacks and driver-assist sensors that learn locally with intermittent links: cars trade models at intersections and keep driving on cached peer models between meetings.
• Smart factories and industrial monitoring where vibration, signal, and scan data stay on the machines: each machine tunes a fault model on its own hum while staying near the fleet consensus.
• Hospital groups combining heart-care, skin-care, and eye-care features into one health index without moving patient rows: specialty knowledge merges through weights, honoring privacy law.
• Banking groups combining branch records, loan-repayment streams, and transaction feeds into one financial index under differing regional rules: same-schema branches average easily while mixed-schema arms need the full caching design.
• Mobile phones, wearables, drones, and vehicles as moving agents that exchange models only inside communication range: on-device typing, health, and navigation models improve with every chance meeting.
• Small edge boards such as Raspberry Pi devices serving as low-cost learning agents with private local datasets: cheap hardware senses and learns at the edge, contributing through stamped cache entries.
• Surveillance cameras and city-scale sensor fleets where centralizing video is impractical, so cached peer models stand in for absent neighbors: each camera keeps watch with the freshest affordable set of peer views.