Quantization, Fine-Tuning, and Low-Rank Adaptation

Slicing a network across devices does not finish the cost story. Every remaining weight and gradient still occupies bits, and those bits still travel on a slow link. This lecture treats three families of cuts: quantization (fewer bits per number), fine-tuning and low-rank adapters (fewer trained numbers), and a short preview of pruning (fewer live weights).

6.1 Communication Bottlenecks After Model and Data Slicing

Distributed training solved two placement problems. It did not make the numbers themselves cheap. The rest of the session attacks that leftover cost.

6.1.1 Recap of Slicing, Mini-Batches, and Pipeline Handoffs

Model slicing (placing different layers, or stage blocks, on different devices) puts stage 1 on GPU 0, stage 2 on GPU 1, and later stages on later GPUs. Data slicing (splitting the training set into batches and then into mini-batches) feeds those stages a stream of small frames rather than one giant tensor. Call a mini-batch a successive data frame if that picture helps: frame 1, then frame 2, then frame 3.

A first mini-batch is processed on the first GPU. That GPU emits activations, the layer outputs. Those activations travel to the next GPU. The next GPU continues the forward pass. The chain repeats until the last stage produces a prediction.

After the last stage, the system computes a loss, a scalar that says how wrong the prediction is. Write $L$ for that scalar. Back propagation then attributes $L$ to each trainable weight. For every mini-batch and every layer, the system computes gradients, the partial derivatives of the loss with respect to the weights.

Those gradients must travel backward along the pipeline so earlier stages can update their weights. Picture a server plus a set of client devices. Each client holds part of the model. After a mini-batch, millions or billions of gradient numbers must move across the interconnect.

Pipeline patterns such as GPipe already showed this handoff: send a mini-batch forward, compute $\partial L/\partial w$ for the local parameters, then send those values back so earlier layers can adjust. GPU 0 processes batch one and sends the output onward. Later, the loss-driven gradients must return.

Think of a bucket brigade at a fire. Water (activations) moves forward from person to person. Empty buckets (gradients) must come back, or the front of the line cannot refill. The analogy breaks because empty buckets are light, while gradients are often as large as the weights themselves.

Two compression targets sit in that picture. You can compress the data representation, how each number is stored. You can compress the model, how many weights you store or update. Both attacks cut storage, arithmetic, and wire traffic.

6.1.2 What Must Be Stored and Moved

A modern network may have millions or billions of parameters. For each parameter you store a weight and you compute a gradient. Every GPU that shares work must exchange those gradients. After mini-batch one finishes on one GPU, the values go across the network to another GPU. That is where a network bottleneck appears: the interconnect is slower than on-device math, and the payload is huge.

The same numbers also occupy GPU memory. A 7 billion parameter model, an 8 billion parameter model, or a 120 billion parameter model makes the problem worse. Pre-trained large language models (LLMs) move enormous tensors during training and during serving. If you work with those models, gradient and weight traffic dominate cost.

A motivating question frames the rest of the material: after slicing, what is the main computational and communication factor you still have to address?

Imagine plotting two curves against model size. The horizontal axis is parameter count, from millions to hundreds of billions. The vertical axis is time per step. On-device multiply time grows, but interconnect time grows at least as fast, because the message is a full gradient tensor. The landmark is the crossing where wire time exceeds math time. Past that landmark, a faster GPU without a smaller message barely helps.

6.1.3 Student Questions and Answers

A natural first guess is that the hard part is still calculus: can we even form $\partial L/\partial w$? That guess is understandable, because back propagation is the engine. It is not the remaining bottleneck.

The correction is sharp. Differentiability is a prerequisite that the training stack already assumes. The new lever is the bit layout of each stored and sent number.

6.1.4 Representation as the Lever, and Today's Map

Computer-architecture courses already show number encodings. FP32 uses 32 bits per value. FP16 uses 16 bits. Integer formats use 8 bits, 4 bits, or 1 bit. The goal is to cut storage and communication overhead without destroying learning.

On a sketch of one 32-bit word, the bits are not equal. Later sections split sign, exponent, and fraction. For now, hold the idea that "a number" is a labeled packet of bits, not a magic real value.

Three families of methods attack that overhead:

1. Quantization changes how each number is encoded, so the same tensor uses fewer bits.
2. Fine-tuning (and later parameter-efficient fine-tuning) changes which weights you update, so you do not train every parameter of a giant model.
3. Pruning later removes or zeros redundant weights and filters. That family is only previewed here.

GPU-to-GPU gradient exchange in pipeline-parallel training is the practical setting. The same bit-width choices also matter when a fine-tuned model must run on an edge device. NVIDIA mixed-precision training, which can keep some values in a 32-bit training format and shrink others to FP16, is the daily form of this lever in large-model stacks.

6.2 Floating-Point Layouts and Integer Encodings

Quantization only makes sense if you know what a "32-bit number" contains. This section opens the packet: sign, exponent, and fraction, then walks the same real value down to 16-bit and integer grids.

6.2.1 Why Bit Layout Matters

A floating-point value is a sign bit, an exponent field, and a mantissa (fraction) field. The sign chooses positive versus negative. The exponent chooses the scale, a power of two. The mantissa chooses the extra fraction bits that sit after an implicit leading 1 in normal numbers.

Think of the layout as a labeled strip of bits. One bit is rose-colored for the sign. Eight bits are green for the exponent in FP32. The remaining yellow bits are the mantissa. Change how many bits each field gets, and you change both the range and the precision.

A postal address is a useful everyday map. The sign is the side of the street. The exponent is the city block (a power-of-two zoom). The mantissa is the house number on that block. The analogy breaks for subnormal numbers and infinities, which use reserved exponent patterns. For normal values the three-field picture is enough.

A first verbal slip in the walkthrough said that a sign bit of 0 means negative. The worked number $-3.14$ then used sign $=1$. Standard IEEE-style encoding uses $s=1$ for negative and $s=0$ for positive. Keep the worked example. Treat the "0 means negative" line as a slip.

6.2.2 Mathematical Formulation of FP32

Let $s\in \{0,1\}$ be the sign bit, $E\in \{0,1,\dots ,255\}$ the 8-bit exponent field read as an unsigned integer, and $M\in \{0,1,\dots ,2^{23}-1\}$ the 23-bit mantissa field read as an unsigned integer. For a normal FP32 value the decoded number is

The spoken formula was: minus one to the power of the sign, times two to the power of (the exponent value minus 127), times one plus the mantissa's decimal value divided by a power of two. One pass said "divided by 2" without the 23. The IEEE layout uses $2^{23}$ because there are 23 stored fraction bits. A denominator of $2$ would only be correct for a 1-bit fraction.

If every exponent bit is 0 except the highest, the 8-bit field is $10000000_2$. Converting bit by bit, each position $k=0,\dots ,7$ contributes $b_k\cdot 2^k$. All zeros cancel. The remaining 1 at position 7 gives $2^7=128$. Then the unbiased scale is

$$\begin{array}{rl}E& =128\\ E-127& =1\\ 2^{E-127}& =2^1=2\end{array}$$

The verbal cue was "this is fixed": you always subtract 127 from the stored exponent in FP32 for normal numbers.

Plot range against fraction bits. The horizontal axis is exponent-field width. The vertical axis is the largest finite $|x|$. FP32 with 8 exponent bits reaches about $2^{127}$. A 5-bit exponent (FP16) reaches only about $2^{15}$. The landmark is overflow: values that fit in FP32 become inf in FP16. That is why some 16-bit formats keep eight exponent bits.

6.2.3 Worked Example: Encoding $-3.14$ in FP32

Setup. Represent $-3.14$. The walkthrough filled three fields and then mixed them with the decode formula.

Sense-check: the result is negative, near $3.14$, and slightly coarse because only 23 fraction bits are stored. That is what you want from a real FP32 decode.

A binary-to-decimal converter is a valid check, but the formula above is the actual decode rule. You do not need to memorize converter websites. You do need the three fields and the bias.

6.2.4 FP16, BF16, and TF32 Variants

FP16 (IEEE binary16) uses 16 bits in total: 1 sign bit, 5 exponent bits, and 10 mantissa bits. The bias is 15, one less than $2^4$ in the same way 127 is one less than $2^7$.

The spoken decode for a 16-bit try on the same $-3.14$ was: minus one to the power of the sign, times two to the power of (the stored exponent minus 15), times one plus $583$ divided by $2^{10}$.

Sense-check: FP16 kept the exponent $2^1=2$, so the value stayed near $3$, but the shorter fraction drifted from $3.140000$ to $3.138672$.

BF16 (bfloat16, spoken as "brain float 16") keeps a 16-bit total but spends the bits differently: 1 sign bit, 8 exponent bits, and 7 mantissa bits. The extra exponent bits keep FP32-like range. The shorter mantissa loses fraction detail.

One cross-check mixed a mantissa integer $72$ with a $2^{10}$ divisor and an 8-bit exponent. That mix collides FP16's 10-bit fraction with BF16's 8-bit exponent. If the integer $72$ is a 7-bit fraction, the correct divisor is $128$:

$$1+\frac{72}{2^7}=1+\frac{72}{128}=1.5625$$

Using $72/2^{10}$ would be the FP16 fraction rule applied to the wrong format.

TF32 (spoken as "TensorFlow 32") was listed as a 19-bit training format used on some GPU and TPU paths. A common industry layout for TF32 is 1 sign bit, 8 exponent bits, and 10 mantissa bits, which sums to 19. NVIDIA's name for that layout is TensorFloat-32. The class label "TensorFlow 32" is the same 19-bit idea: keep FP32's exponent width and FP16's fraction width.

$$x_{TF32}=(-1{)}^s\cdot 2^{E_8-127}\cdot \left(1+\frac{M_{10}}{2^{10}}\right)$$

Different vendors pick different default formats for GPU versus TPU training. Conversion among those formats is why quantization is not a side trick. It is the daily number-format layer of large-model training.

6.2.5 Integer Eight-Bit Range and Four-Bit Collapse

Signed INT8 does not store a hidden-1 fraction. It stores an integer in about 8 bits. One teaching picture uses 1 sign bit and 7 magnitude bits. The positive magnitudes $1,2,4,8,16,32,64$ sum to $127$. The stated integer range to map into is $[-128,+127]$.

Integer values such as $127$ can land exactly. The pain starts when you force a float with a rich mantissa into a small integer grid.

A 4-bit signed picture uses 1 sign bit and 3 data bits. The maximum positive magnitude is $7$.

Sense-check: 4-bit signed integers cannot represent $8.978$ at all, so the collapse toward $7$ is not a small mantissa trim. It is a range failure.

As you walk from 32 bits toward 1 bit, representation error grows. If the true value is $-3.14$ and INT8 stores $-3$, the gap is already $0.14$. If that gap is tiny next to the task, you may accept it. If you are updating weights, even a small mantissa can matter, because convergence depends on precise increments.

6.2.6 Student Questions and Answers

The $2^7$ prompt is the exponent-field conversion used in the $-3.14$ decode.

GPU and TPU vendors expose these layouts as native types. Tensor cores prefer FP16, BF16, or TF32 packets. A weight that looks like "3.14" in Python is already one of these bit strips in hardware.

6.3 Quantization of Weights and Gradients

Quantization means rewriting a number in a smaller encoding. This section defines the send path first: quantize, ship, and optionally dequantize. Memory counts and the INT8 integer map come after that path.

6.3.1 Definition, Pipeline, and Trade-Off

A gradient may start in FP32 or FP16. You quantize it, send the short packet over the network, and, if you still want a floating value for the optimizer, you dequantize on the far side.

The pipeline in words is: take the gradient in some format such as 32 or 16, make a lesser-size representation, send it, then optionally expand it again so you can update weights. Dequantization tries to recover a usable float. It cannot invent bits that were never sent.

Think of a photo sent as a tiny JPEG. The file is light. The far side can expand it to a full-size image. The expansion cannot restore faces that the JPEG already smeared. The analogy breaks because JPEG is nonlinear, while the class INT8 map is a multiply, a round, and a divide.

Benefits named in class:

• lower communication cost
• less GPU memory
• faster processing of each tensor
• lower network bandwidth
• quicker training in wall-clock time

The cost is information loss. Moving from 32 bits toward 1 bit, or toward 4 bits, throws away fraction bits. Weight updates that needed a mantissa like the fractional part of $32.6$ will not see that extra precision. The result is a trade-off: smaller storage and smaller messages, versus weaker updates and possible harm to convergence. You must measure whether the lost bits still let the model learn.

6.3.2 Memory and Bandwidth for a 7 Billion Parameter Model

A concrete memory picture used a 7 billion parameter model during a training step that holds FP32 weights. The spoken phrase was that you need 28 GB if you want the weight update in that representation.

Each FP32 value uses 4 bytes. Then

Sense-check: 7 times 4 is 28, and "billion parameters times bytes" is "gigabytes" in the decimal GB used in class.

FP16 cuts the payload in half. The 16-bit layout was described as 1 sign bit, then 5 exponent bits, then 10 mantissa bits. Bandwidth saving is $2\times$. Memory for the same 7 billion weights is 14 GB. The INT8 footprint is the next rung.

INT8 uses 8 bits per value, spoken as 1 sign bit plus 7 remaining bits. Relative to 32 bits you keep $8/32=1/4$ of the bits, so you save a factor of 4. One sentence also said "8 times you saved." The bit ratio $32/8=4$ is the consistent memory ratio, and 7 GB was the stated INT8 footprint. The "8 times" line matches a 32-to-4-bit cut (INT4), not INT8. Keep 7 GB as the INT8 figure.

$$M_{FP16}=14\mathrm{GB},M_{INT8}=7\mathrm{GB}$$

INT4 and 1-bit encodings were listed on the same ladder. Each further cut saves more memory and more wire time, and each further cut grows error. You cannot look only at the saving column.

Accuracy comments, given as rules of thumb rather than a named benchmark:

• FP32: no accuracy loss from format, maximum bandwidth
• FP16: accuracy change called negligible, bandwidth saving $2\times$
• INT8: a drop "less than one percent" in a generic story, with an immediate warning that 1% can be huge for a hard target
• INT4: used when you must deploy to edge devices or ship a fine-tuned model in a federated setting; a 1% to 2% drop may be acceptable

T1 reports the same qualitative trade: mixed precision often still converges, but it can land in a worse local minimum than full precision, because values such as $1$ and $1.01$ can collapse to the same code.

A GPU face detector at 99% accuracy may still be valid on a Raspberry Pi after INT4 if the drop is about 2% and the bandwidth saving is large. One spoken saving figure was "eight percent," which does not match an 8-times bit cut. Treat the Pi story as "small accuracy drop, large resource saving." The reliable bit-ratio for INT4 versus FP32 is $32/4=8$, not an 8% bandwidth cut.

6.3.3 Where Quantization Sits in the Training Loop

The painful moment is back propagation, not a quiet forward pass on a laptop. Workers hold FP32 data and FP32 gradients. For each weight they compute a gradient. Those tensors are quantized. The compressed values travel through the network to the destination GPU. On arrival you may dequantize if you want more numeric range inside the optimizer. You may also keep the quantized values if the optimizer can use them.

In distributed training the exchange is staged. GPU 1 sends gradients to GPU 2, GPU 2 to GPU 3, GPU 3 to GPU 4, in the reverse direction of the forward pipeline. After the forward pass you compute gradients and exchange them so each stage can adjust its weights. Using 8-bit values instead of 32-bit values shrinks that reverse traffic to one fourth, so communication becomes faster. Storage falls. Arithmetic can fall. Precision also falls. That is the same trade-off again.

GPipe-style mini-batching still applies. Data is divided into mini-frames. Mini-batch one enters the first stage, activations flow forward, the last stage computes loss, and $\partial L/\partial w$ for each block must go back. Those returning tensors are the values you consider for quantization.

Sketch the pipeline on a page. The horizontal axis is stage index (GPU 0 to GPU 3). The vertical axis is time. Forward activations step down-right. Backward gradients step down-left. The landmark is the backward diagonal: that is the fat payload quantization tries to thin.

6.3.4 INT8 Scaling Formula and Worked Example for $0.65$

Floating formats had a field-wise IEEE-style formula. Integer quantization uses a scale that maps a real range onto the integer grid.

Setup. Take the real value $x=0.65$. Choose the signed INT8 grid $[-128,+127]$. Use the maximum positive integer $127$ as the scale, the "maximum possible one" in the spoken walkthrough.

Sense-check: $83/127$ is a hair above $0.65$, because $82.55$ rounded up. Rounding down to $82$ would have sat a hair below. The grid cannot hit $0.65$ exactly.

The verbal pair is: quantization is multiply-by-scale-and-round; dequantization is divide-the-integer-by-the-same-scale. You "apply the reverse" of the formula.

You must then ask whether $0.003543$ per value disturbs learning and convergence. For one weight it looks tiny. Across a billion weights, the same class of error can pile up. The residual will accumulate: each step adds a small grid error, and billions of weights multiply the chance that those errors push updates the wrong way.

6.3.5 Student Questions and Answers

The first question is the decode step itself.

Lost bits are a separate confusion. Expanding an integer does not recreate a deleted fraction.

A third instinct is to stay in integers inside the original network math.

6.3.6 When to Insist on Dequantization

Quantize-then-dequantize is not mandatory for every net. For complex detection, including object detection and real-time accident detection, keep the extra care: quantize for the wire, dequantize for the update, and test whether the task still converges. For simpler CNN detection, you may not need that extra expand step.

Edge and federated deployments often force a small integer format such as INT4. A 1% to 2% drop can be acceptable if the device constraint is real. A GPU at 99% face-detection accuracy versus a Raspberry Pi with about a 2% drop is the concrete "still valid" story.

6.3.7 Industry Applications and Practical Checks

GPU and TPU vendors expose FP32, TF32, BF16, FP16, and integer paths. Conversion among those encodings is daily work in large-model training, not a homework toy. NVIDIA Automatic Mixed Precision is one library form of the same ladder: wrap an optimizer and let the runtime pick FP16 where it dares.

Interactive converters let you type any number and watch FP32, FP16, BF16, and integer reconstructions. Integer inputs are the easy case. The hard case is a float whose mantissa does not survive the smaller grid.

6.4 Fine-Tuning and Transfer Learning

Quantization left trained weights in place and only shortened their encoding. This section asks a different question: how do you teach a giant pretrained model a company task when you cannot retrain billions of entries?

6.4.1 Fine-Tuning Is Not Quantization

A prompt asked how fine-tuning differs from quantization.

Quantization leaves the trained weights in place and only shortens their encoding. You do not retrain just to quantize. Fine-tuning updates model weights by training on your own data. The aim is to turn a generic model into a specialist for your task and your company data. A large language model never saw that private corpus, so you must introduce the corpus if you want outputs that match it.

Fine-tuning a 120 billion or 8 billion parameter net from scratch is not realistic. Those models were trained by labs such as Meta and Google on terabytes of general text and images. You still want to inject custom images, for example defective MRF tires, into a vision-language model (VLM). Named VLMs in the discussion included Qwen (spoken as QUN / Kwan), Mistral (spoken as mystical), and a Llama-family model (spoken as polygama / polygamma). The three product families are the intended set.

Without a parameter-saving trick, "fine-tune from scratch" would mean retraining billions of weights. That path is closed.

6.4.2 Domain Adaptation With Few Company Images

A VLM is a vision-language model: it jointly handles images and text. It can caption an image, or generate an image from a caption, and it can act as a visual agent on mixed inputs including video. Those models are strong generalists because they were trained on huge multimodal corpora.

They are not born as MRF tire inspectors. Defects may be a vertical scratch, a horizontal scratch, or other marks on the tread or sidewall. You may have only about 100 labeled company images, not millions. Training a detector from zero on 100 images will not match a foundation VLM. So you adapt the pretrained model.

Domain adaptation here means: keep the general knowledge already baked into billions of parameters, and add a small amount of company-specific knowledge so the model becomes a task specialist. After adaptation, the same VLM becomes an MRF tire defect expert. It can mark defects on new images and it can run faster because the serving path is specialized, not a giant general chat model doing a side job.

The data scale named for the company trial was hundreds of samples, not millions. With that small set, Qwen, Llama-family, and Mistral-style models were trained until a new image produced a defect decision.

Think of the pretrained weights as a world encyclopedia. You do not rewrite the encyclopedia. You add a thin company pamphlet and staple it on. The analogy breaks if the pamphlet contradicts the encyclopedia on facts that must stay frozen; then you are in catastrophic forgetting, which is flagged later, not solved here.

6.4.3 Layer Freezing, Feature Extraction, Partial and Full Fine-Tuning

A first family of tricks is old transfer learning.

T1's layer-freezing chapter adds a systems reading of the same freeze: once a pipeline stage has converged, you can drop its activations and skip its gradient work, which frees GPU memory during model-parallel training. That is freeze-for-memory. The class story here is freeze-for-adaptation: keep generic features, train the task head.

Sense-check: 1539 is a tiny trainable set next to a ResNet or a VLM. That is the point of freezing.

The verbal contrast is sharp. On ResNet-scale models, partial or even full fine-tuning can be fine. BERT-base fine-tuning on a single GPU is the same size class. On ChatGPT-scale models, updating all layers or even a large last block is not possible in this setting. That is why the discussion turns to PEFT and LoRA.

So the simple stack is: freeze a lot, train a little, maybe walk the trainable window deeper. For giant LLMs and VLMs you still need a more extreme cut: parameter-efficient fine-tuning (PEFT), including LoRA.

6.4.4 Student Questions and Answers

The opening contrast is the one to keep in exam language.

A second instinct already points at PEFT.

Company defect inspection with a handful of images is the reason PEFT exists. You cannot collect ImageNet-scale labels inside one factory.

6.5 Parameter-Efficient Fine-Tuning and LoRA

Chat-scale models made full updates impossible. Parameter-efficient fine-tuning (PEFT) attacks that wall. Low-rank adaptation (LoRA) is the PEFT method developed in detail: freeze $W$, train two thin factors, add their product.

6.5.1 Split a Giant Matrix Into Two Thin Matrices

Start with a square weight matrix of size $n\times n$. For a mental picture, take $n=1000$. Then

$$n\times n=1000\times 1000=1,000,000$$

which is 10 lakh parameters in one matrix. LoRA does not update that full grid. It writes the grid as a product of two much thinner matrices whose inner dimension is a small rank $r$.

Keep the row count $n$. Choose a tiny column count $r$, such as $2$, $4$, or $8$. One factor is $n\times r$. The other is $r\times n$. Their product is again $n\times n$:

Forget a long linear-algebra sermon about rank if you need to. The working picture is: pick a small $r$, split, multiply back to the old shape, and train only the split pieces.

A sliding door on a wide wall is a fair everyday map. The wall (frozen $W$) already stands. You add a thin pair of rails (the two factors) that let the door move a little. The analogy breaks if $r$ is so small that the door can only slide in one direction; then the adapter cannot express a rich task.

6.5.2 Mathematical Formulation

Let $W\in {\mathbb{R}}^{d\times d}$ be a frozen pretrained weight matrix. Let $r\ll d$ be the LoRA rank. Introduce two learnable matrices. One spoken assignment was $A\in {\mathbb{R}}^{d\times r}$ and $B\in {\mathbb{R}}^{r\times d}$. Another later line swapped the names and said $A$ is $r\times d$ and $B$ is $d\times r$. The product must be $d\times d$ either way.

This write-up fixes the names so that $BA$ is $d\times d$: $B\in {\mathbb{R}}^{d\times r}$ and $A\in {\mathbb{R}}^{r\times d}$. If you swap the letters, swap the product order too. Do not mix the two conventions in one line.

Forward pass, including a scaling factor $\alpha /r$ (spoken both as "alpha by R" and, in one pass, as if it were a learning-rate term):

$$y=Wx+\frac{\alpha }{r}(BA)x$$

Here $x\in {\mathbb{R}}^d$ is the layer input, $y\in {\mathbb{R}}^d$ is the layer output, $\alpha$ is a positive scale hyperparameter, and $r$ is the rank. The frozen path $Wx$ stays. The adapter path is the only path you update. $\alpha /r$ is not the optimizer learning rate. It is a separate knob that keeps adapter updates a similar size when you change $r$.

Before training, $A$ and $B$ are initialized to small random values so that $\Delta W$ starts as a small change to $W$. A common library default sets $B$ to zero and $A$ to a small Gaussian, which makes $\Delta W=0$ at step 0. The class version (both factors small and random) is the same idea: start with a tiny pamphlet, not a second encyclopedia.

After training, $\Delta W$ encodes domain-specific patterns: legal phrasing, tire-scratch textures, or whatever the new task is.

One wording said predictions use only $\Delta W$ and not $W$. The forward formula above still includes $Wx$. The intended claim is that updates use only $A$ and $B$, while the frozen $W$ still contributes to the forward pass.

6.5.3 Worked Examples of Parameter Counts

Train skinny factors, then multiply back to $n\times n$. The counts below are the exam arithmetic.

Sense-check: $400/10,000=0.04$, so you train 4% of that matrix. The multiply shape is unchanged.

The $4\times 4$ toy is the same pattern with numbers you can write on one board.

Sense-check: a full $4\times 4$ has 16 entries too, so rank $2$ on a $4\times 4$ is not a saving. The toy is for shape, not for compression. Compression appears when $r\ll d$.

Example D — million-scale motivation. A $1000\times 1000$ matrix is 10 lakh entries. LoRA replaces that with two skinny factors whose inner size is a small $r$. That is the point of the method: keep the multiply shape, drop the trainable count.

The small numbers inside $A$ and $B$ are the task-specific weights. After they learn, you "assign back": add $BA$ into the working model so the fine-tuned net contains the new knowledge.

6.5.4 Back Propagation Through $A$ and $B$

For each training example the model predicts, you compute a loss against the ground truth, then back propagate. You do not update $W$. Gradients flow into $B$ and $A$, which together form $\Delta W$.

Let $L$ be the loss and let $h$ be the layer output after the adapted multiply,

$$h=Wx+\frac{\alpha }{r}BAx$$

Let $g=\partial L/\partial h\in {\mathbb{R}}^d$, a column. Chain rule in the spoken sketch was "dL by dH, times dH by the next factor," then a gradient involving the transpose of $A$.

After many examples, $A$ and $B$ settle. Their product encodes the task. Adding that product to $W$ yields the fine-tuned matrix. The values you add are small. A spoken numeric cartoon was: if a frozen entry is $20$ and the adapter entry is $0.03$, the sum is $20.03$. Some adapter entries stay 0.

Loss may be classification or regression. The model predicts $\hat{y}$ as a function of the adapted weights. You compare to the label, form $L$, and send $\partial L/\partial A$ and $\partial L/\partial B$. Repeat. That is ordinary back propagation on a tiny variable set, with the giant $W$ frozen.

6.5.5 How Many Parameters You Train, and Optimizer-State Saving

You can vary $d$, $r$, and how many layers get LoRA. Full fine-tuning trains the whole matrix. LoRA trains $2dr$ per adapted matrix. One comparison said that after LoRA you might train only "93 lakh" parameters against a much larger full count. The matching full-model denominator was not given, so treat 93 lakh (9.3 million) as a named worksheet figure, not a universal ratio.

A per-layer worksheet used:

• full matrix parameter count on the order of $3.07\times 10^x$ in a display that repeated 3.07 (the exact power was not clean)
• LoRA $A$ with shape involving $r\times d$
• LoRA $B$ with shape involving $d\times r$
• LoRA total per layer = both factors, times the number of adapted layers
• compression ratio from those counts
• scale $\alpha /r$

A named saving was 95.3% of optimizer states relative to the dense baseline. Optimizer states (moment buffers in Adam-style methods) track each trainable tensor. If you train 400 numbers instead of 10,000, those extra buffers shrink with the trainable set.

A training simulator compared feature extraction against LoRA. Feature extraction showed a wide gap between training loss and validation loss, read as "it will not learn much" and as overfit. LoRA reduced that gap and moved more useful information into the updated adapter weights. The simulator is a way to see that a small trainable set can still fit the new task.

Transfer learning with a frozen backbone is "very simple" next to LoRA. The piece to learn deeply is PEFT/LoRA: choose two low-rank matrices, train them, freeze $W$, add the product.

Catastrophic forgetting, the risk that new training erases old skills, is flagged as a later topic, not derived here.

6.5.6 Student Questions and Answers

Fine-tuning is not a distributed-only trick.

A compression instinct from linear algebra does not replace LoRA.

LoRA is how teams attach legal, medical, or factory knowledge to a frozen foundation model without renting a full pretraining cluster.

6.6 Fine-Tuning Versus RAG and Prompting

LoRA changed weights. A common confusion is to treat that change as the same trick as retrieval-augmented generation (RAG). This section splits the tools: external lookup versus baked-in skill.

6.6.1 Retrieval-Augmented Generation Is External Memory

RAG is a storage and retrieval process. It sits outside the model. Documents live in a database. A user question arrives. The language model receives that question and checks whether matching data lives in the RAG store. If a match exists, the system pulls the corresponding context. Now the model holds two pieces: the user question and the retrieved context. It writes an answer from both.

You can dump PDFs, presentation decks, Jira tickets, Confluence pages, or medical notes into that store. Text question-answering is the sweet spot.

Fine-tuning changes the model. Task knowledge is inside $A$ and $B$, then inside $W+BA$. There is no extra fetch step at inference for that knowledge. You imbibe the new skill into the weights themselves.

You can also stay outside the weights with prompts only: a zero-shot prompt or a few-shot prompt (spoken as "few short" / "zero short"). That path never updates $W$. It only shows extra text or extra example images in the request.

When to pick which: retrieve text that already exists; train when the model must do a new perceptual or stylistic skill.

6.6.2 Why Tire Bounding Boxes Do Not Belong in RAG

Suppose the task is: given a tire image, draw a bounding box on a scratch. The expectation is a box on the damaged region, not a paragraph of advice. RAG will not match that job. You cannot store a few defect photos in a text database and expect a retrieved paragraph to localize pixels. Storing the image in RAG and hoping a match appears does not give you coordinates.

Prompting was tried without touching weights: show an "upper line defect," a "vertical defect," and then a new photo. It did not work for this complexity. The team had to show images during training so the model became an expert. For complex problems the model may not understand the prompt exemplars. For simple fetch-and-read jobs, such as reading PDF documents, prompts may help.

The split in one sentence: RAG is for textual documents you insert and retrieve. Fine-tuning is for skills you must bake into weights, such as drawing boxes on factory images. Object detection for a specific company, such as MRF tires, is the fine-tuning side. Document QA is the RAG side.

6.6.3 Generic Answers Versus a Domain Model

A generic model, asked whether a California employer may fire a worker for a social-media post made outside work hours, tends to hedge: employment law varies by state, exceptions apply, here is a vague pointer. A domain-tuned model can cite a labor code, name exceptions, and give a tighter reason. The same pattern is the MRF detector: general VLM versus tire expert.

Picture two desks. On the left, a generalist reads a question and hedges. On the right, a specialist has already studied the code and answers with a citation. RAG can slide a statute onto the generalist's desk at ask-time. Fine-tuning is sending the specialist to school before the question arrives.

6.6.4 Student Questions and Answers

The mix-up is to treat LoRA as another name for RAG.

Jira and Confluence dumps are RAG workloads. Factory camera inspection is a fine-tuning workload. Mixing the two tools without that split is a design error.

6.7 Preview of Pruning

Quantization shortened bits. LoRA shrank the trainable set. Pruning is the third compression family, scheduled for a later session. The preview is enough to separate it from quantization and LoRA.

6.7.1 Zeroing Weights Instead of Retraining Everything

Pruning adjusts the weight tensor by forcing some values to zero, as if you snipped branches. If a weight is $18.3$ and it fails a keep-test, you store $0$. You can also drop filters, which cuts the number of live kernels in a convolutional layer. The aim is fewer effective weights, less storage, and less training and serving cost.

T1 describes a related serving cut as model distillation via pruning: define a keep-test, drop neurons that fail it, and keep a smaller net with the same job. That full recipe is later. Here the picture is already enough: zero what you do not need.

Dropout is named as one pruning-style technique: it reduces the number of active weights, though classical dropout is stochastic at training time. The later session goes deeper.

6.7.2 Do Not Delete Layers; Collapse Redundant Filters

A convolutional layer is not itself the learned object. The filters are. A spoken shape "34 comma 3" means 34 filters of size $3\times 3$. Then the layer learns

Sense-check: 34 small $3\times 3$ patches have $34\times 9=306$ spatial weights. That is a board-sized count, not a 7 billion parameter count. The lesson is where learning lives, not the absolute size.

If you declare 64 or 128 filters, several of them may learn nearly the same feature. The distance between two filter matrices can be tiny. If two filters look for the same pattern, you can keep one copy. That merge is one pruning style. Declared width 128 does not mean 128 distinct concepts. Near-duplicate filters are the ones you remove to cut training time and storage.

Sketch two $3\times 3$ grids that look almost equal: both light up on a vertical edge. The keep-test notices the tiny distance and drops one grid. The layer still exists. It just has 127 filters instead of 128.

6.7.3 Student Questions and Answers

The branch picture is the right spirit, if you zero weights rather than delete stages.

The dangerous shortcut is to remove a whole layer.

Over-wide CNNs with 128 filters often duplicate detectors. Collapsing those duplicates is a practical serving optimization, not only a diagram in a textbook.

Exam Guidance Summary

No numeric mark map was given. The emphasis signals still tell you what to practice.

• After pipeline slicing, the remaining cost is storing and shipping gradients. Bit-width is the first lever.
• Know the FP32 split: 1 sign bit, 8 exponent bits, 23 mantissa bits, bias 127. Replay $-3.14$ at least through sign, exponent integer 128, and $2^{128-127}$.
• Know that FP16 uses bias 15 and 10 mantissa bits, and that the same $-3.14$ became about $-3.13$ in the 16-bit try.
• Know signed INT8 range $[-128,+127]$ and why 7 magnitude bits top out at 127.
• Memorize the INT8 scale template: $q=\mathrm{round}(x\cdot 127)$ for $x=0.65$ gives 83; dequantization is $83/127$; error is $0.003543$. Write every intermediate, not only the final float.
• Pair the 7 billion parameter memory ladder with the accuracy warnings: 28 GB FP32, 14 GB FP16, 7 GB INT8, then INT4/1-bit with growing error.
• Dequantization does not restore deleted bits. Integer-only sends are legal and error accumulates across billions of weights.
• Fine-tuning is not quantization. Freeze-and-train-a-head is the simple path. Full updates are for ResNet-scale nets, not ChatGPT-scale nets.
• LoRA is the PEFT method to know: $W^{\prime }=W+BA$ with $r\ll d$, train $A,B$ only, optional scale $\alpha /r$. Replay the $100\times 100\to 400$ count.
• RAG is external retrieval. It is not LoRA. Bounding-box inspection needs fine-tuning. PDF/Jira/Confluence QA can use RAG. Few-shot prompts failed on the hard visual task.
• Pruning preview: do not delete CNN layers; merge similar filters; dropout was named as related. Details later, along with catastrophic forgetting.
• Interactive number converters and a LoRA training simulator were recommended as practice, not as a substitute for the formulas.

Key Industry Applications

• Pipeline-parallel GPU training (GPipe-style): activations go forward, quantized gradients come back. Bit-width cuts message size on the reverse path.
• Foundation LLM/VLM serving: GPT-style, BERT-style, Qwen, Mistral, and Llama-family models are too large to retrain; PEFT/LoRA is the practical adaptation path.
• Factory visual inspection: MRF-style tire defect detection with about 100–hundreds of images, needing bounding boxes, not a document index.
• Edge and federated deployment: INT4 (and similar) on devices such as a Raspberry Pi, accepting about a 1–2% accuracy drop to win memory and bandwidth.
• Accelerator formats: FP32, TF32 (19-bit), BF16, FP16, INT8 chosen by GPU/TPU vendors; conversion among them is quantization work. NVIDIA mixed precision is the common training form of that conversion.
• Enterprise text stacks: RAG over PDFs, Jira, and Confluence for lookup; not a replacement for visual fine-tuning.
• Legal specialists: a generic model hedges on California employment questions; a tuned model cites labor code.
• CNN serving: prune near-duplicate filters in wide layers (64 or 128 kernels) to cut storage and time.
• Transfer baselines: ResNet-style full or partial fine-tuning still works when the backbone is small enough; giant chat models need LoRA.
• Optimizers: LoRA's 95.3% optimizer-state saving matters because Adam-style buffers scale with the trainable tensor count, not with the frozen $W$.