BERT, GPT, and Vision-Language Models

16.1 Exam Preparation Road Map

16.1.1 What the Comprehensive Exam Will Cover

Before the last content block of the course, here is the full picture of what the final exam rewards. Treat these six points as a checklist — each one is a lever you control before exam day.

First, review the past comprehensive question papers together with their posted solutions. The style of those papers is the strongest predictor of what you will see again: same question families, similar numerical structures, comparable mark distribution. Working a past paper under time pressure and then checking the posted solution tells you two different things — where you lose marks for knowledge and where you lose marks for speed.

Second, questions related to Assignment 2 are guaranteed to appear. If you worked the assignment diligently and also studied the theory behind each step, you are covered there. The distinction matters: an examiner can take any step you executed in the assignment and ask "why this step, and what would happen without it?" — that version cannot be answered from memory of your submission alone.

Third, bring a scientific calculator. Programmable calculators are not expected; follow whatever the official communication says about the allowed model. Diffusion and energy-based questions can involve evaluating exponentials, so a calculator you already know how to drive saves real minutes.

Fourth, refresh the deep neural network basics you already know: weight updates using backpropagation, and parameter calculation of a network. You are expected to remember these without re-learning them. For parameter calculation, practice counting weights and biases layer by layer on small networks until it is mechanical.

Fifth, pre-midsem topics occupy only a small slice of the paper — about 5 to 15 percent, not more. The rest comes from everything covered after that point plus the recommended reading. Allocate study hours roughly in that proportion.

Sixth, three areas are named explicitly as likely question sources: diffusion plus energy-based models, natural language processing (NLP) applications, and vision applications — alongside the Assignment 2 topics. Notice how much of this lecture feeds directly into those areas: BERT and GPT are the NLP applications; captioning and visual question answering are the vision applications.

16.1.2 Study Plan and Practical Advice

Practice numerical problems. You can generate extra drills with ChatGPT or any similar tool, focusing on diffusion basics, energy-based models (EBMs), NLP applications, and the assignment topics. A useful drill loop: ask for a problem, solve it fully by hand, then ask for the worked solution and compare line by line — the mismatches show exactly which concept needs another pass.

Printed watermarked slides may be carried into the exam room, so prepare them in advance. Well-organised printouts work as a second memory: annotate them during revision so finding a formula mid-exam takes seconds rather than minutes.

Managed well, you have more than ten days of preparation time — enough to cover all of the above comfortably. A simple split works: first pass over post-midsem topics, daily numerical practice from day two onward, full past papers near the end, and light slide-annotation throughout.

16.2 BERT: Bidirectional Transformers for Language Understanding

16.2.1 Why Bidirectionality Matters: Understanding versus Generation

The architecture to master here is BERT — Bidirectional Encoder Representations from Transformers, built at Google. (The name is sometimes misheard as "BART"; every property listed next — masking, next-sentence prediction, an encoder stack, and the Google origin — identifies the model unambiguously as BERT.) The phrase "language understanding" is the key part of its name, because deep understanding powers an entire family of applications:

• Sentiment analysis — deciding whether a text expresses a positive or negative opinion ("This phone is a beautiful paperweight" — positive words, negative sentiment; catching that needs real understanding).
• Question answering — answering a question from a given passage, by finding where in the passage the answer lives.
• A large family of similar tasks: paraphrase detection, entailment, entity recognition.

Why does BERT look at text in both directions? Compare it with a generative model. To generate new text, an autoregressive (AR) model may only condition on tokens that already happened — the past. There is no choice about this: you cannot generate a word from words that do not exist yet. If you are writing "The cat sat on the ___", the next word must be produced using only "The cat sat on the". But BERT's scope is understanding, not generation, so this restriction disappears. When the goal is to interpret a word in a sentence, using the tokens on its right is perfectly reasonable — indeed, the right-side context is often exactly the evidence you need.

Real-world placement: this understanding-first design is why BERT-style models sit behind search engines (matching a query to relevant documents), sentiment systems, and question answering products, rather than behind text generators.

16.2.2 Architecture Properties

Three properties distinguish the architecture.

1. Depth. It stacks multiple layers of transformers, so it is a deep architecture. Each transformer layer receives information from a position, from positions before it, and from positions after it — every layer sees both sides. Nowhere in the stack is there a rule limiting information to the past. (Contrast this with the decoder's masked self-attention, which zeroes out attention to future positions precisely so that generation cannot cheat.)

2. Bidirectionality, precisely stated. The representation of any token is computed from the full surrounding context, left and right. This is the precise meaning of "bidirectional" here: not two separate networks reading opposite ways, but every token attending to every other token.

3. Self-supervised pre-training. Like many models in this course, it uses self-supervised learning to build its initial language competence. During pre-training the model learns to predict a held-out token from all the remaining tokens — predict token $T_2$ from $T_1$ together with $T_3$ through the end of the sequence. No human labels are needed for this stage, which is what unlocks enormous corpora.

16.2.3 Pre-training Task 1: Masked Language Modelling

Take a sentence such as "The cat sat on the mat." For every sentence, randomly pick some words and replace each with a dummy mask token. About 15 percent of the input tokens get masked. The model must predict each masked word from the tokens before it and after it — both sides feed the prediction through the transformer weights.

Formalizing the goal. In words, the training goal reads: maximise the probability of the masked word given all the other tokens, changing all the parameters of the network to achieve it. Written as an objective:

$$\hat{\theta }=\arg \underset{\theta }{\max }{P}_{\theta }\left(w_{\text{mask}}\right|{\mathbf{w}}_{\text{context}})$$

Each symbol in turn:

• $w_{\text{mask}}$ is the hidden token — the word replaced by the mask.
• ${\mathbf{w}}_{\text{context}}$ is the set of all surviving tokens around it, from both the left and the right side.
• $\theta$ collects every trainable parameter of the architecture — attention weights, feed-forward layers, embeddings, all of it.
• $\arg {\max }_{\theta }$ means "the value of $\theta$ that makes the quantity after it as large as possible."

Because the context includes tokens from both directions, the procedure forces the model to fuse information from the whole neighbourhood of the mask into one representation. This task is completely unsupervised — really self-supervised, since the labels come from hiding parts of the input itself.

In practice the maximisation runs as gradient descent on the equivalent minimisation: take the negative logarithm of that probability and drive it down. At each masked position the model outputs a score for every vocabulary word, a softmax turns scores into probabilities, and a multiclass cross-entropy loss compares the predicted distribution with the true hidden word. Maximising $P_{\theta }(w_{\text{mask}}\mid {\mathbf{w}}_{\text{context}})$ and minimising $-\log P_{\theta }(w_{\text{mask}}\mid {\mathbf{w}}_{\text{context}})$ are the same instruction.

Worked sketch: if the original sentence is "cat sat on the mat" and the mask lands on "cat", the model sees the embedded representations of everything else and adjusts all parameters until $P(\text{cat}\mid \text{context})$ is high. Repeat over millions of sentences and the encoder absorbs grammar, facts, and word senses.

A note on efficiency worth knowing: MLM makes inefficient use of data. Only about 15 percent of positions contribute a prediction, so processing seven tokens adds roughly two loss terms. Autoregressive models predict at every position and use data more efficiently — but they give up the right-side context. That trade-off is the whole design decision between encoder and decoder.

The original token embeddings may come from fixed encodings such as GloVe vectors before training starts; the MLM objective then sharpens them into contextualised representations — the same word ends up with different output vectors in different sentences, which a fixed embedding can never do.

16.2.4 Pre-training Task 2: Next Sentence Prediction

BERT pre-training has a second, equally particular innovation: next sentence prediction (NSP). Given a sentence A during pre-training, with 50 percent probability you attach the truly following sentence as segment B; for the remaining 50 percent you attach a random sentence that does not follow A. A classification label rides along with every pair:

$$y_{\text{NSP}}=\{\begin{array}{ll}1& \text{if segment B really follows segment A}\\ 0& \text{if segment B is a random sentence}\end{array}$$

The model reads both segments and predicts the label. So indirectly it learns what makes one sentence follow another — discourse coherence. The pre-training corpus supplies unlabelled sentence pairs in bulk; the 50/50 split keeps the task honest, since always guessing "yes" would score only half.

Together, MLM and NSP are the complete pre-training recipe. Both run on a huge unlabelled corpus, and both shape the contextualised token representations that fine-tuning later exploits. (Worth knowing for perspective: later analysis found the NSP signal contributes only marginally once MLM is strong — a fact RoBERTa will act on in the next section.)

16.2.5 Input Representation and Tokenization

Many NLP tasks feed two segments at once: a question and an answer passage, or two sentences plus a paraphrase/entailment label. BERT's input format reflects this. Consider the pair "my dog is cute" and "he likes playing".

Subword tokenization. The text is first broken into tokens by a highly optimised subword tokenizer — rare or composite words are split into smaller pieces so the model never faces an unknown word. BERT's scheme is called WordPiece: a vocabulary of frequent pieces (whole common words plus frequent fragments like "##ing") covers any input by composition. So "unhappiness" might split into "un" + "##happiness", and even a typo or a made-up word still maps to known pieces. WordPiece is the standard subword scheme shipped with BERT.

Special tokens. A special classification token [CLS] opens the sequence and a separator token [SEP] divides the two segments. The [CLS] position will later carry a summary vector used for whole-sequence classifications.

Three summed embeddings. On top of raw tokens the model computes three kinds of embeddings, summed into one vector per position:

• Token embeddings say which subword sits there.
• Position embeddings say where it sits in the concatenation of both segments — the position counter runs continuously across the pair, not restarting at each segment.
• Segment embeddings say which of the two segments it belongs to — one embedding value for segment A, another for segment B.

Position:    1      2    3   4    5       6     7    8     9        10
    Token:    [CLS]   my   dog  is cute  [SEP]  he  likes playing  [SEP]
    Segment:    A     A    A    A    A       A     B    B      B        B

A practical warning: the embeddings BERT produces are sensitive to the tokenizer choice. The WordPiece vocabulary shipped with the model was tuned together with it, so swapping tokenizers degrades the representations — always load the matching tokenizer when reusing a checkpoint.

16.2.6 Fine-Tuning for Question Answering

After pre-training comes fine-tuning on a specific task with limited labelled data. Question answering is the canonical example. The input is the question as one segment and the paragraph that contains the answer as the other segment.

Two output vectors are attached at the top, one for answer start and one for answer end:

$$\mathbf{s},\mathbf{e}\in \{0,1{\}}^N$$

where $N$ is the number of paragraph tokens, $\mathbf{s}$ carries a 1 at the token where the answer starts, and $\mathbf{e}$ carries a 1 at the token where it ends — all other entries are 0. Supervised training pushes the two vectors toward these one-hot targets, so the network learns to classify the boundary tokens of the answer span inside the passage. In implementation terms, every token position produces two scores (start-score and end-score); softmax over the paragraph positions turns each set of scores into a distribution, and training maximises the probability mass on the true boundary tokens.

This is why the workflow is called semi-supervised: the bulk of the learning is self-supervised pre-training on a huge corpus, and only a thin supervised layer is added per task. The same pre-trained checkpoint fine-tunes into question answering, summarisation-style tasks, and other understanding jobs.

16.2.7 Where Masking Came From: The Cloze Task

The historical echo is pleasing: a seventy-year-old psychology instrument became, with transformers behind it, one of the most productive training objectives ever found. The mechanism transfers because filling a blank well requires exactly what we want the model to learn — grammar, semantics, and world knowledge fused into one prediction.

16.2.8 Training Cost and Industrial Scale

BERT came out of Google, which owns enormous training corpora and its own accelerator hardware: the TPU (Tensor Processing Unit), a chip designed specifically for the matrix multiplications that dominate neural network training. Pre-training runs on TPU-based servers and takes on the order of days. Nobody retrains daily — the schedule is more like once a year or once in a few months. Fine-tuning, in contrast, is cheap: only a few hours, because it nudges an already competent representation rather than building one from scratch.

BERT reached state-of-the-art results on language understanding benchmarks: paraphrase detection, sentiment analysis, question answering. Its limit is equally important: it has no generation capability. Everything it does rests on deep understanding of language — ask it to continue a document and there is simply no machinery for producing the next token from past-only context.

Recap: BERT is a deep bidirectional transformer encoder trained self-supervisively with MLM and NSP, then fine-tuned cheaply per task — built for understanding, not generation. The natural next question is whether the recipe itself can be improved without touching the architecture, which is exactly RoBERTa's story.

16.3 RoBERTa: An Optimized BERT

16.3.1 What Changed in Pre-training

An improved version followed: RoBERTa — the name expands to a robustly optimized BERT. Most of the engineering carries over unchanged. Three changes to the pre-training strategy deliver the gains.

First, the next sentence prediction task is dropped entirely — it turned out to be unnecessary overhead. Removing it frees model capacity and training time for the task that actually drives understanding, MLM, and simplifies the input format since single sentences no longer need pairing.

Second, the masking strategy changes: instead of masking a single static choice of locations per sentence, masking covers multiple locations, refreshed during training. In static masking, each sentence is pre-masked once and the model sees the same blanks every epoch — it can quietly memorise those particular completions. Dynamic masking re-draws the mask pattern each time a sentence is fed in, so across epochs the same sentence yields many different prediction problems, which acts like free data augmentation and keeps the task from going stale.

Third, the model trains on an even bigger corpus, scaling up masked language modelling itself — more data, larger batches, longer training. The gains came from doing the same objective harder, not differently.

16.3.2 What Stayed the Same

The architecture remains a bidirectional transformer encoder, the objective remains masked language modelling, and the deploy pattern remains pre-train then fine-tune. Nothing in the network drawing changed at all — same attention blocks, same encoder stack, same embedding scheme.

Recap: RoBERTa kept BERT's body and rebuilt its training schedule, proving that recipe quality was being left on the table. Next we flip to the other family entirely — models built to generate rather than understand.

16.4 BERT versus GPT: Understanding versus Generation

16.4.1 Encoder versus Decoder

Set the two families side by side.

Same underlying transformer machinery, opposite information regimes, opposite jobs. The masked self-attention inside a decoder is what enforces the past-only rule: attention weights to future positions are set to zero before the softmax, so no position can peek ahead during training or generation.

Earlier standard practice built a separate model from scratch for every downstream task, including older sequence-to-sequence systems for language generation. GPT-1 changed that: pre-train one language model, then apply a standardised fine-tuning procedure across all downstream tasks. One backbone, many heads — the same transfer-learning move BERT made on the understanding side.

16.4.2 Stock Transformers: The Analogy

Think of what pre-training buys with a stock-photography analogy. A stock photo library holds pictures of all kinds — landscapes, offices, people mid-handshake; a buyer picks and adapts rather than staging a shoot from scratch. A stock text corpus likewise holds a sample of the language itself: books, articles, forums — language as it is actually used. Pre-training learns the language model embedded inside that stock of text — the transformers internalise how the language works, its grammar, its common associations, its typical phrasings. Downstream, you adapt this stock representation to your specific need: a sentiment head here, a question-answering span predictor there.

Where the analogy breaks: a photo buyer only ever re-crops or recolours what exists, whereas a fine-tuner changes the model itself — and a prompt-steered GPT does not even do that much. The deeper point survives both readings: as long as huge training data exists, you can learn the language model to a very large extent, and the learned model unlocks generation, question answering, and many fascinating capabilities.

16.5 GPT-2 and GPT-3: Scale and Prompting

16.5.1 Scaling Parameters and Context

GPT-2 and GPT-3 extend the original GPT. Two things distinguish them: the scale of the architecture and the scale of the context used to generate.

On sizes: GPT-2 sits around the billion-parameter range (roughly 1.5 billion), and GPT-3 reaches about 175 billion parameters, trained on roughly 300 billion tokens. (If you ever meet a figure like "more than 150 million parameters" for GPT-3, read the unit as billions; the published size agrees once the unit is corrected.) Either way the trend is what matters: each generation is orders of magnitude larger than the last. Both remain completely unsupervised — self-supervised language modelling, no task labels at all. Every token of training text supplies its own supervision: predict the next token from the ones before it.

The deeper change is procedural, not architectural. GPT-1 needed a fine-tuning stage with labelled data for each downstream task. GPT-2 and GPT-3 drop task-specific fine-tuning. Instead, the prompt itself steers the model — you describe or show the job inside the input text, and nothing about the weights changes.

16.5.2 Zero-Shot Learning with Prompts

Zero-shot learning means solving a task from a single instruction prompt, with no training examples attached. The prompt acts as a cue that reshapes how the pre-trained network behaves — a very shallow adaptation compared with gradient-based fine-tuning, where labelled data physically moves millions of weights. Here no weight moves at all; only the input pattern changes.

This raised an immediate doubt in class, and the resolution matters:

Why did the RAG guess seem plausible? Because if a model answers questions without training on them, something must be supplying the missing knowledge — and retrieval is one honest way to do that. The resolution is subtler: the knowledge was already inside the pre-trained network, compressed there during self-supervised training on the huge corpus. The prompt does not deliver knowledge; it selects which of the behaviours already learned should surface. The takeaway: the prompt is not a database lookup. It is a demonstration folded into the input, and the network's pre-trained knowledge does the rest.

16.5.3 Few-Shot Learning in GPT-3

Few-shot learning packs a handful of solved examples into the prompt before the real query.

2 + 3 = 5
    9 + 8 = 17
    12 + 30 = 42
    3 + 4 =

The same pattern works for editing requests: hand over three corrected-writing demonstrations — original page, corrected page — then supply a fresh page to fix, and the model returns the corrected page by imitating the shown transformation. The demonstrations define the mapping; the model applies it.

16.5.4 Data and Compute Requirements

The key message cuts both ways. With enormous training data and nearly limitless computing, you can learn the language model almost completely. Generation, question answering, summarisation, and design-flavoured generation all fall out of it. Remove those resources and you fall back on traditional NLP techniques — hand-built parsers, task-specific feature engineering, smaller statistical models.

Real-world placement: companies with vast data and compute — Amazon, Google, OpenAI — can digitise data themselves, train, and keep improving such models; scale is their moat. For small datasets on local infrastructure, the realistic hope is open-source models; several strong ones now come out of China and compete respectably with proprietary systems such as GPT and Gemini. The strategic question for any team is not "which architecture?" but "whose checkpoint can we afford to adapt?"

Recap: GPT-2/3 scaled parameters and context, then replaced fine-tuning with prompts — instructions for zero-shot, demonstrations for few-shot. Having followed the generation branch to its conclusion, we now switch modalities: what happens when images join text?

16.6 Vision-Language Tasks: What Modern Systems Can Do

16.6.1 Captioning, Visual Question Answering, and Retrieval

Given one image, today's AI systems produce a caption — a natural-language description of the scene (visual captioning). Given an image plus a question, they answer it: visual question answering (VQA). Ask "What is the woman doing?" and the answer comes back "riding a horse". Counting questions work too: how many horses are there, how many women.

Retrieval closes the loop in the opposite direction: submit "show me horses on which a woman rider is present in front of a river" and the system fetches matching images from a database. Notice what that query demands — the system must parse a compound spatial-and-identity condition ("horse", plus "woman rider", plus "in front of river") and score every image against all three clauses at once.

These are joint-modelling applications: the system must represent image content and textual content together, and connect them. Neither modality alone is enough — the text describes only what the image shows, and the image only matters through what the words say about it.

16.6.2 The Joint Tokenization Principle

There, you tokenise the corpus into word pieces, pre-train, and optionally fine-tune for understanding or generation. Here, you tokenise the text as before — and tokenise the image as well, turning pixels or detected regions into discrete units a transformer can consume alongside words. Joint modeling of image tokens with text tokens enables the tasks above: captioning attends from image tokens to generate words; VQA conditions answers on both; retrieval scores image-text pairs with one shared representation.

Processing demand grows enormously compared with text alone, especially with state-of-the-art transformers stacked in multiple layers over both modalities — two modalities mean more tokens per example, longer sequences, and cross-attention patterns to learn on top of each modality's internal structure. The technical recipe extends what this course already covered; cost and purchase constraints are set aside here.

Recap: captioning, VQA, and retrieval all reduce to one recipe — joint token streams from both modalities. The next section starts with the simplest classical way to build such a bridge, before any transformer enters the picture.

16.7 Classical Visual Captioning with CNN plus RNN

16.7.1 Pipeline Walkthrough

Basic visual captioning needs no transformer. It is supervised learning applied in a smart way, using pieces from a standard deep learning course.

Purpose. The problem: map an image to a word sequence that describes it. Two different data types must meet — an image grid of pixels and a sentence of discrete words — and something has to translate between them.

Inputs and outputs. Input: an image plus its training caption (image-caption pairs). Output at generation time: the caption, one word at a time.

The whole system is a joint CNN-plus-RNN architecture trained end to end on caption pairs. Its view of the image is global: one vector stands for the entire picture.

Complexity and cost. One CNN pass per image plus a number of RNN steps equal to caption length — cheap compared with transformers. But the single-vector bottleneck limits expressiveness: a 512- or 1024-number summary must carry everything worth describing, so fine spatial relationships tend to wash out.

16.7.2 Ways to Improve the Basic System

Several upgrade paths exist:

• Swap in a different CNN architecture for stronger vision features — better backbone, better description of what is present.
• Replace the plain RNN with an LSTM variant to capture long-term dependencies across longer captions — gates inside the LSTM decide what to remember and what to forget, which keeps early words ("a woman") available when generating later ones ("who is juggling").
• Add attention mechanisms so the decoder can focus on relevant image regions while emitting each word — instead of one global vector, the decoder consults a weighted mix of region features, re-weighted per word.
• Bolt transformer-based language modelling on top of the CNN features as a stronger word-prediction engine.

Each upgrade raises caption quality, but the global-feature bottleneck remains until the region-based methods of the coming sections — those replace "one vector per picture" with "one token per object", which is precisely what joint modelling needs.

Recap and bridge: CNN encodes, RNN decodes, captions supervise. The limitation — everything squeezed through one vector — motivates both the self-supervised tricks coming next and, eventually, token-per-region representations.

16.8 Self-Supervised Learning for Images

16.8.1 Image Colorization

For image colorization, take photos captured with a mobile phone and manufacture the input-output pair yourself — no human labeller ever enters the loop. Create the grey version by simple averaging of the colour channels:

$$I_{\text{gray}}=\frac{R+G+B}{3}$$

where $R$, $G$, and $B$ are the red, green, and blue channel values of each pixel (each in $[0,255]$ for 8-bit images) — the sum of the three channels divided by three. A note on standards: many image tools instead use a luminance-weighted blend such as $0.299R+0.587G+0.114B$, which mimics eye sensitivity; the plain average is what this lecture assumes.

Train on the pair (grey input, colour original) and the model learns colorization: given greyness at every pixel, predict the three missing channel values per pixel.

Beyond this global version, chop the image into small $16\times 16$ blocks and colourize per patch. Patch-wise processing lets the model specialise to local textures — grass needs one palette, brick another — because each patch presents a narrower, more consistent prediction problem than the whole scene at once.

16.8.2 Image Inpainting

Image inpainting follows the masked-language blueprint even more directly. From one original photo, generate several masked copies — punch holes or cover regions, differently each time. Train a generative model to reconstruct the complete image from the masked input. Suitable engines include GAN variants and the diffusion models covered earlier in the course.

Recap and bridge: colourization hides chroma, inpainting hides regions; both turn unlabelled photos into free supervision. But real vision-language systems need more than repaired pixels — they need captions, and captions live on the internet, noise included, which is exactly where we go next.

16.9 Pre-training on Large, Noisy, Cheap Data

16.9.1 Weak Labels from Social Media

Vision-language pre-training consumes large, noisy, cheap data: uncurated image-plus-text pairs already lying around in social media posts, digitised books, journals, and magazines. No collection campaign is needed — the data exists whether or not anyone curates it, and that is what makes it cheap.

Alongside this raw data you need some small, clean, labelled data for fine-tuning. End users often supply the labels unknowingly: comments and likes on social media images act as implicit annotations, for example signalling positive or negative sentiment toward an image. A million likes is a million weak votes for "this image pleases people". Companies such as Google exploit exactly this user-generated labelling at scale — the users are the annotation workforce without ever being told.

16.9.2 Case Study: A Mismatched Caption and Hallucination

Train on many such pairs and the model absorbs spurious associations — it overtrains on words the pixels never justify. The mechanism: gradient descent does not know which words are grounded and which are decoration; minimising caption likelihood rewards producing "Thailand"-flavoured text whenever the visual context resembles these photos. Consequence: hallucination. If a later image shows two children in a similar background, the system may emit a confident caption referencing Thailand or unseen intentions, and the answer looks nonsensical to the user.

This is why hallucinations observed in modern image models often trace to imperfect training data rather than to the architecture alone. The network faithfully learned exactly what we taught it — including our errors.

16.9.3 The Standard Strategy: Pre-train, Then Fine-tune

Because uncurated pre-training can never be enough (unless captions were curated perfectly, which they never are), the typical strategy is two-stage:

1. Stage one — foundation model. Learn a foundation model on stock data of the noisy kind shown above — the broad base model that everything else adapts, with the name emphasising its role as common groundwork. The goal is broad coverage of vision-language statistics, accepting some noise as the price of scale.
2. Stage two — fine-tune. Adapt that model on small clean labelled data for each downstream task. Clean labels here re-anchor the model to grounded associations after the noisy pass.

Visual question answering illustrates the boundary. Reasonable questions — who is in the image, what are they doing, urban or rural setting — receive sound answers after fine-tuning. Hyper-specific questions about caption content the image never supported return hallucinated answers, because the model's confidence was calibrated on ungrounded text.

Recap and bridge: noisy data gives scale, clean data gives grounding, and hallucination names what happens when the first runs without the second. Next: how to represent images so they can sit beside words inside one transformer at all.

16.10 From Pixels to Tokens: Region-Based Visual Tokenization

16.10.1 Bounding Boxes as Visual Tokens

Modern systems no longer squeeze an image into one global feature — the bottleneck of the classical captioner. Instead they tokenise the image itself using region-based CNN detectors: the R-CNN family, including Faster R-CNN, from the computer vision toolkit. A YOLO-style detector serves the same role (YOLO stands for You Only Look Once, a one-pass object detector). Detection places a bounding box around each object and classifies its content.

16.10.2 Learning Spatial Semantics

Joint training over visual and textual tokens teaches surprisingly abstract language. Three mechanisms do the teaching:

• Relative positions carry spatial words. Seeing the couch box below and the person box above teaches the model what "on" denotes — the geometry of the purple sofa region under the red person region instantiates the preposition. Nobody wrote a definition of "on"; thousands of stacked configurations are the definition.
• Co-occurrence binds synonyms. The same visual configuration labelled sometimes sofa, sometimes couch, ties the two words together — they compete to describe identical pixels, so their embeddings converge.
• Association attaches attributes. Linking the red person region with the word man in text lets the model infer gender attributes from vision plus language jointly — the word anchors onto the region across many images until the pairing is reliable.

Capturing such cross-token associations is exactly what attention in a transformer does naturally — every token can consult every other token — which is why attention-based joint encoders dominate this area.

Recap and bridge: regions become tokens; geometry becomes meaning; attention does the wiring. Next question: should the two modalities be encoded separately and merged late, or together from the start?

16.11 Two-Stream versus Single-Stream Fusion

16.11.1 Late Fusion: Separate Modality Encoders

Early vision-language architectures used two streams:

1. The sentence — say "a young man playing frisbee" — passes through its own self-attention transformer, learning a contextualised representation of the text modality.
2. The region tokens from an R-CNN or YOLO-style detector pass through a second self-attention transformer, learning a contextualised representation of the image modality.
3. A third transformer then merges the two contextualised streams.

Each modality first becomes fluent in its own language, and only then do the languages meet.

The weakness is where fusion happens: high and late. If the text says "young person" instead of "young man", aligning the word with the corresponding visual region becomes harder, because the two modalities only meet after each has already frozen its own representation. The text encoder committed to a person-without-gender reading; the image encoder committed to region summaries with no knowledge that a gender word might matter — and only the third transformer must now reconcile both commitments from above.

16.11.2 Early Fusion: One Shared Transformer

Later architectures went single-stream: feed text tokens and image tokens together into one multilayer transformer, so cross-modal relationships form at lower levels. From layer 1 onward, the token for man can attend directly to the red person-region token; alignment is built up gradually inside every layer rather than patched on at the end. This is early fusion, set against the two-stream late fusion.

Recap and bridge: fuse early, let every layer negotiate between words and regions. The next section opens up UNITER's pre-training objectives — the full masking toolkit applied to this single-stream design.

16.12 UNITER: Joint Pre-training Objectives

16.12.1 Embedding Setup and Notation

UNITER is a high-performing single-stream joint model — one shared transformer over words and regions, exactly the early-fusion design the last section settled on. Its pre-training shows all the masking ideas in one place.

Embeddings follow the BERT pattern extended to vision:

• Image side: region content embedding plus a location encoding computed from the normalised bounding-box coordinates, passed through fully connected layers and layer normalisation. The location encoding is what lets geometry participate at all — without it, two boxes with identical content but different positions would be indistinguishable, and spatial words could never ground.
• Text side: token embeddings and position embeddings, concatenated, then layer normalisation.

Everything trains jointly — one parameter set θ serves all objectives defined below.

16.12.2 Masked Language Modelling with Visual Context

Task one applies the masking principle uniformly — the model does not fuss over which tokens are textual and which are visual; masking just hides items and demands them back.

For a masked word, the goal in words: maximise the log likelihood of the masked word given all the other text tokens and all the visual region tokens, adjusting the parameters theta of the whole network. Written as a minimisation-ready loss:

$${\mathcal{L}}_{\text{MLM}}(\theta )=-\sum _{m\in {\mathcal{M}}_w}\log P_{\theta }\left(w_m\right|{\mathbf{w}}_{\setminus \mathcal{M}},v_{1:K})$$

Each symbol: ${\mathcal{M}}_w$ selects the masked word positions; ${\mathbf{w}}_{\setminus \mathcal{M}}$ is the surviving text (everything except masked slots); $v_{1:K}$ means all image regions $v_1$ through $v_K$; the minus sign converts maximising likelihood into minimising loss. This reconstruction follows the verbal description "maximise the log likelihood ... with respect to all words and all vision region tokens by changing the parameters theta" — and it passes its sanity checks: the sum yields a scalar loss, and setting $K=0$ collapses it back to plain BERT-style MLM.

The difference from pure BERT is precisely the conditioning: visual evidence helps pick the word. If the sentence reads "the man rides a [MASK]" and the image contains a horse-shaped box under the rider, the probability mass shifts toward "horse" over "bicycle" — information no amount of left-right text alone supplies reliably.

16.12.3 Masked Region Modelling: Regression and Classification

Task two masks image regions instead of words — and it splits in two, because a region carries more than content: it also owns a bounding box.

Part A — masked region feature regression. From the fused hidden states, regress the coordinates of the masked region's box. In words: use regression over the bounding boxes, so that minimising the regression functional makes the model detect bounding boxes correctly:

$${\hat{\mathbf{b}}}_m=g_{\theta }\left({\mathbf{H}}^{\mathrm{VM}}\right),\underset{\theta }{\min }{\ell }_{\text{reg}}({\hat{\mathbf{b}}}_m,{\mathbf{b}}_m)$$

Symbols: ${\mathbf{H}}^{\mathrm{VM}}$ denotes the fused vision-language hidden states (the transformer's output vectors); ${\hat{\mathbf{b}}}_m$ is the predicted four-number box for masked region $m$, produced by a small head $g_{\theta }$; ${\mathbf{b}}_m$ is the ground-truth box; ${\ell }_{\text{reg}}$ is a coordinate regression loss, with standard squared-error on the normalised coordinates being the usual reading. Domain check: because coordinates are normalised to $[0,1]$, a good prediction must land inside the unit square — anything outside is visibly wrong.

To see why this forces real understanding, trace the logic: the region's own features are hidden, so the only route to guessing where it sat is context — the words describe it ("the mug on the table") and the surrounding boxes constrain it (between the table top and the shelf). Geometry must be inferred from neighbours, exactly the skill spatial language requires.

Part B — masked region classification. Given all the words and all vision tokens except the masked $m$-th region, predict that region's object class. Training computes the cross entropy between the predicted class distribution and the correct class label. Additionally, a KL divergence compares the predicted content distribution with the ground-truth content distribution, checking that the contents agree. In words from the lecture: calculate the cross entropy, and use KL divergence to decide whether the content here and the content there are the same.

$${\mathcal{L}}_{\text{MRC}}(\theta )=\mathrm{CE}(P_{\theta }(c_m),c_m^{\star })+D_{\mathrm{KL}}(P_{\theta }(\cdot )\Vert P_{\text{target}}(\cdot ))$$

with $c_m^{\star }$ the true class of masked region $m$, $P_{\theta }(\cdot )$ the predicted class distribution, and $P_{\text{target}}(\cdot )$ the target distribution. The relative weighting between the cross-entropy and KL terms was not stated in the session, so they are shown unweighted here — the two terms play complementary roles: cross entropy rewards putting probability on the right single class, while the KL term disciplines the whole predicted distribution against a reference, penalising confident disagreement anywhere, not just at the winner.

Why mask regions at all? Same reason as masking words: prediction under hiding forces fusion. To classify an invisible region, the model must combine textual clues ("a shaggy dog") with visible neighbours (a leash leading off-frame), which trains exactly the cross-modal wiring these systems exist for.

16.12.4 Image-Text Matching

Task three asks a blunt question: do this image and this sentence belong together? A classification head on the CLS-style token outputs

$${\hat{y}}_{\text{ITM}}\in \{0,1\}$$

where 1 means the region tokens and sentence tokens match and 0 means they do not. Pair the boss-asleep image with an unrelated sentence and the label is 0.

The mechanism leans on the same CLS-and-pair design that BERT's next sentence prediction made famous, lifted from sentence pairs to image-sentence pairs.

16.12.5 Combined Loss Function

Pre-training optimises all three objectives together. In words: the loss function is made up of three things — image-text matching, masked region estimation, and masked language estimation — and you want to optimise all of them together.

$${\mathcal{L}}_{\text{pre}}(\theta )={\mathcal{L}}_{\text{MLM}}(\theta )+{\mathcal{L}}_{\text{MRM}}(\theta )+{\mathcal{L}}_{\text{ITM}}(\theta )$$

where ${\mathcal{L}}_{\text{MRM}}$ bundles the regression and classification parts of masked region modelling from the previous subsection. Shape check: each term is a scalar, so their sum is a scalar — one number gradient descent can chase.

The purpose of the whole stage: train every transformer weight so that words can consult regions and regions can consult words. Recap and bridge: pre-training has now wired the joint representation; the next section walks through the downstream tasks that fine-tuning plugs into it.

16.13 Fine-Tuning Tasks for Vision-Language Models

16.13.1 Visual Question Answering

Fine-tuning plugs the pre-trained joint model into concrete tasks — first of all VQA. The pre-trained weights already know how words and regions interact; fine-tuning only teaches the output format of each new job.

Sample questions show the difficulty ceiling:

• "What colour are her eyes?" (answer: black) — needs locating a face, then a sub-part of it.
• "What is the moustache made up of?" — needs material-level recognition, not just object labels.
• Counting demands genuine perception: "How many slices does this pizza have?" requires detecting the cut boundaries that separate slices and then counting them — you cannot count slices without first seeing where one ends and the next begins.
• Judging whether the pizza is purely vegetarian or a mix of vegetarian and non-vegetarian toppings requires fine-level content identification at topping granularity.

Trickier items probe inference beyond direct perception: "Is this person expecting company?", "What is just under the 3?", "Does it appear to be rainy?", "Does this person have 20/20 vision?". The model breaks the question into pieces, grounds them in image tokens, and answers — sometimes from pixels alone ("rainy": umbrellas, wet ground), sometimes by combining pixels with commonsense.

16.13.2 Visual Entailment

Here the premise is the image and the hypothesis is a caption; the model judges one of three labels: entailment, neutral, or contradiction.

A standing caveat ties back to tokenization: if the original visual tokens coming out of the detector are poor, entailment judgements inherit the errors — wrong person-class detections flip entailments into false contradictions. That failure mode belongs to the R-CNN stage, not to the language modelling.

16.13.3 Natural Language for Visual Reasoning

This task hands the model two images and one compound statement, and it answers true or false.

16.13.4 Visual Commonsense Reasoning

Beyond perception, the model must reason about motives. Worked example: "Why is person 4 pointing at person 1?" Among candidate rationales, the correct one: person 3 is delivering to the table and might not know whose order this is, so person 4 points to clarify.

Selecting that answer requires fusing the visual scene — who holds what, who faces whom — with social commonsense supplied by language: pointing near food service conventionally means clarifying an order. Neither modality alone licenses the rationale; vision gives the configuration, language gives the social script.

16.13.5 Referring Expression Comprehension

Given a referring phrase such as "the woman watching dishes", the model must locate the referred person among all the people in the image. It is grounding in its purest form: language picks one entity out of many visual candidates, using the distinguishing clause ("watching dishes") as the filter. Output is a box on the right person — retrieval inside a single image rather than across a database.

16.13.6 Image-Text Retrieval

Retrieval runs both ways: given text, retrieve matching images; given an image, retrieve matching descriptions. A query like "a guard with a cat on grass" is scored against an image database using the CLS-style classifier — each image-query pair gets a true/false match score, and the highest scorers are returned.

This completes the syllabus. Optional supplementary material exists beyond this point, but it is not part of the syllabus.

Exam Guidance Summary

• Review past comprehensive question papers and their posted solutions; expect similar question patterns, with additions from Assignment 2. Working one paper under time pressure before checking solutions shows you both knowledge gaps and speed gaps.
• Assignment 2 topics are guaranteed exam material; know the underlying theory, not just the submitted answers — examiners can ask "why this step?" about any part of it.
• Bring a scientific calculator; programmable calculators are ruled out by official communication.
• Refresh deep neural network basics: weight updates using backpropagation and parameter calculation of networks. These are assumed known, not re-taught.
• Pre-midsem topics contribute only about 5 to 15 percent of the paper; allocate revision hours accordingly.
• Core expected areas: diffusion plus energy-based models, NLP applications (BERT and GPT families), vision and vision-language applications, and Assignment 2 topics.
• Practice numerical problems; tools like ChatGPT can generate extra drills on diffusion basics, EBMs, NLP applications, and assignment topics — solve by hand first, then compare against its worked solution.
• Printed watermarked slides are allowed in the exam room; arrange printouts early and annotate them during revision so formulas can be found in seconds.
• Time budget: more than ten days of preparation if managed well — enough for a full post-midsem pass, daily numericals, and past papers at the end.

Key Industry Applications

• Sentiment analysis and question answering built on bidirectional understanding models (BERT family) — the engines behind search relevance, support-ticket routing, and review mining.
• Text generation driven by decoder models (GPT family), with zero-shot and few-shot prompting replacing task-specific fine-tuning — one deployed model now serves drafting, summarising, and editing through different prompts.
• Periodic large-scale pre-training on TPUs at Google; cheap downstream fine-tuning as the deployment pattern — pre-train once a year or so, adapt many times daily.
• Social media likes and comments harvested as weak labels for vision-language fine-tuning data — users annotate at scale without ever filling a form.
• Image captioning and visual question answering in consumer AI products — accessibility describers for visually impaired users, photo-search assistants.
• Image colorization and inpainting for photo restoration, powered by GANs and diffusion models — reviving archives and retouching images with zero per-photo labelling cost.
• Cross-modal retrieval: searching image databases with text queries and vice versa — stock photography search, e-commerce "find this product from a photo".
• The compute-and-data moat of Amazon, Google, and OpenAI versus the rising open-source alternative ecosystem, including competitive models out of China — the practical choice facing most teams is whose checkpoint to fine-tune, not whether to train from scratch.