Cross-Lingual and Multimedia Information Retrieval

This lecture covers two extensions of basic search. The first half builds cross-lingual retrieval, where the query language and the document language differ, starting from query and document translation choices through dictionary cohesion to parallel-text models with IBM Model 1 and expectation maximization, and closing with comparable corpora and multilingual merging. The second half moves from text only to multimedia retrieval, covering scope, the semantic gap, metadata, piggyback, annotation and fingerprint search, then content-based image retrieval with cosine similarity and color histogram comparison.

15.1 Cross-Lingual and Multilingual Definitions

15.1.1 What Cross-Lingual Retrieval Means

What if you ask in English but the best answer was written in French?

A cross-lingual information retrieval system (how much a query in one language finds documents in another language), written with query language $L_q$ and document language $L_d$ where $L_q\ne L_d$, lets a person pose a query in $L_q$ and get back documents written in $L_d$. A multilingual information retrieval system (how much one query searches documents in many languages at once), written with one $L_q$ and a set $\{L_1,L_2,\dots ,L_k\}$ of document languages, builds on the same base and adds merging across languages. The base task is cross-lingual. The added task in multilingual work is how to merge ranked lists from many languages into one display for the end user.

So one query language plus one different document language is cross-lingual. One query language plus documents in many languages is multilingual. The travel of information from source form to target form is shared. The difference is the count of languages handled and the merge step.

Think of a post office that only reads English addresses. Letters arrive with French, Italian, and Korean addresses. Without a translator at the counter, the sorter cannot route them, even though the delivery trucks work well. The translator does not deliver letters. The translator makes addresses readable so the existing sorter can work. Translation in retrieval plays that counter role. It does not replace ranking. It makes ranking possible across languages.

A side-by-side contrast helps fix terms:

When to pick which framing: use cross-lingual when the target store uses one foreign language. Use multilingual when one feed must mix many languages, such as a news aggregator pulling one event from many countries.

15.1.2 Motivating Examples From Everyday Search

Common events make the need concrete. Take a sports query in English only about long jump, winners, list of participants, and how events around it unfolded. News about such an event appears in many countries and many languages. A news aggregator that pulls those different-language documents for one feed is doing multilingual work. No single-language crawl covers the story. Each country reports its own athletes, its own times, and its own side events. Only a system that crosses languages can join them.

The same holds for a pandemic. During COVID-19, guidance, case reports, and news appeared in different languages in different places. A single English query that pulls that spread is useful only if the system can cross the language gap. Early guidance from one region, case counts from a second region, and research notes from a third region all matter at once. Waiting for human translation of each page loses days. A cross-lingual pass finds candidates fast, and humans then check the top few with care.

A second concrete anchor: web search where a person in another country asks in English and wants content that was first written in Spanish, then read in English. The query language and the document language differ by design, not by error. The system must treat that as the normal case, not as a corner case.

Picture a bar chart of web content by language. The x-axis lists languages with page counts. The y-axis shows number of pages. English holds a large bar, but the combined height of all other bars is larger still. The takeaway: a query locked to one language can only reach one bar while the rest of the store stays dark.

15.1.3 Student Questions and Answers

Real-world and domain placement in one place: web search engines use this daily when an English query typed abroad returns French, Italian, or Korean pages. News aggregators use it to join long-jump results, participant lists, and side-event reports across languages. Public-health teams used it during COVID-19 to pull guidance and case reports written in many languages. In each case the retrieval core stays the same, and the bridge decides whether the right foreign documents can score at all.

15.2 Reasons to Build Cross-Lingual Retrieval

15.2.1 The Corpus Is Not in Our Language

Why build a bridge at all? Because the store holds answers we cannot read yet.

A source language (the language of the query as issued), written $L_s$, and a target language (the language of the documents to be retrieved), written $L_t$, name the two ends. The user may have zero knowledge of the target language and cannot search it directly. The goal is that anyone who needs information can get it, in any language where it was written.

Two small facts make this urgent. First, much new content appears first in a local language, not in English. Second, older valuable stores (patents, manuals, legal files, health notices) were written for local readers. A reader who waits for an English version may wait forever. Retrieval must go to the text as written.

Picture a pie chart of a mixed store. One slice is English. The other slices are Korean, Hindi, French, Italian, and more. One-sentence takeaway: a query that only matches the English slice leaves most of the pie unread.

15.2.2 Societal Economic and Crisis Uses

Three families of need show why the bridge matters beyond convenience.

Societal benefit comes from official guidance. Health guidance published by a state is often published in the local language only. A citizen who reads guidance from Karnataka, Delhi, or Andhra Pradesh benefits when retrieval crosses into those languages. Dose schedules, clinic lists, and quarantine rules are written for local readers first. An English-only search misses them. A cross-lingual pass finds them, and a short translated snippet is enough to act.

Economic benefit comes from patents. When a patent is filed in South Korea, most details such as title and approval details appear in Korean only, unless an English title was added on purpose. To claim a competitive edge, a filer must know what patents already exist in that domain. Only a filing that is not already present can become a new patent. That check needs retrieval across Korean and English. Missing one Korean filing can sink a whole application, so recall matters more than polish here.

Crisis response is the most pressing case. During an earthquake or a pandemic, field reports come from agents on the ground. Those agents do not write polished English in distress. They write in local languages because lives are counted in minutes, not retrieval times. Cross-lingual retrieval lets those reports be found by people who do not read the local language. Speed beats style. A rough match now is worth more than a perfect match tomorrow.

15.2.3 Passive Knowledge Factoids and Pre-Selection

Passive knowledge (the ability to read a language better than one can write it), scored here as read-level skill minus write-level skill, motivates a large family of uses. A person who reads Telugu script can often read Kannada script too, even when writing Kannada feels hard and full understanding is partial. That person can still try to read documents in Kannada while issuing queries in English. The same pattern holds in Europe, where basic school French lets a reader get through French documents. India is a strong example. A reader of Hindi can often read and partly follow several other North Indian languages. A reader of Telugu can often follow Kannada. A reader of Tamil can often follow Malayalam. That partial reading ability makes retrieval across scripts valuable even without full fluency. The user does not need a perfect translation. A ranked list with readable snippets is enough to recognize the right document.

Think of riding a bicycle with training wheels. Full fluency is riding without help. Passive knowledge is riding with light support: the reader wobbles on grammar but still moves forward on meaning. The support here is the ranked list plus short translated titles. Where the analogy breaks: training wheels never change the road, while a ranked list changes which roads the reader even sees, so ranking quality still controls what passive reading can reach.

Factoids (stable facts such as names, places, dates, and chemical formulas that change little across languages), for example $H_{2}O$ or 1947 or Marie Curie, give a useful anchor. Cross-lingual methods can lean on those anchors because they carry over. Names and formulas survive translation with little shift, so they act like fixed points while surrounding words move.

Pre-selection (narrowing a huge set to a small set for careful human work), for example cutting 10,000 legal documents to 10, is another use. Suppose 10,000 legal documents must be checked. No team can hire human translators to render all of them into the source or target language. The cost is too high. Even an imperfect cross-lingual pass that cuts 10,000 to 10 changes the task. A legal translator then renders only those 10.

Everyday anchors make the same point in media form. Hotel booking in Italy through a third-party tool shows reviews at the bottom that decide the booking. Some reviews are in English and some are in Italian. The Italian reviews are often the most honest. Without cross-lingual help a traveler misses them and may book a poor hotel. Many viewers watch a Malayalam crime thriller in its native language with captions. The caption bridge is the same idea in media form: keep the source as is, add a light bridge, and let the viewer judge.

15.3 Where Translation Sits in Retrieval

15.3.1 Query Translation Versus Document Translation

How do an English query and a French store ever meet?

The setup gives two vector spaces. The query lives in a vector space of English terms. The document lives in a vector space of French terms. Both are vectors, but their term axes differ. To score overlap we must bring them to shared ground. Here foreign, written $F$, means the non-English side, and English is written $E$.

Option one brings all foreign documents, where foreign here means the non-English side, into English, then uses the English query to retrieve English versions. Here foreign is marked as $F$ and English as $E$. The verbal description kept is "bring documents in foreign language to English, then use English query".

Option two keeps documents as they are and maps the query into the foreign language, then retrieves foreign documents. The verbal description kept is "translate query to foreign language and retrieve documents in foreign language".

Think of moving house across a river. Query translation ferries a small bag (5 words) across. Document translation ferries the whole house (hundreds of documents, each 500 to 600 words) across before anyone visits. The bag is cheap to move but holds little context. The house is costly to move but keeps every room. Where the analogy breaks: a house stays moved once ferried, while translation quality decays with time as language and collections change, so even the costly move needs refresh.

15.3.2 The Translation Module as the Single Point of Failure

The block shape is simple. A query in a source language and a document in a target language meet at a translation block. The whole crux lies in getting that translation right. The source can be the query or the document. The target can be the query or the document. Either way the work boils down to a correct translation model between source and target.

The lower part stays familiar: representations for query and document, a similarity measure, ranking, retrieval, and feedback. That part is the same as in monolingual work studied across many sessions. Nothing new happens below the bridge. Tokenize, weight, score, sort, and refine with feedback as before.

A tiny number sketch shows the failure. Suppose the English query term bank should map to French banque (money bank) but maps instead to rive (river bank). The ranker then scores documents with $score=$ overlap against rive. Truly relevant documents with banque get $score\approx 0$. Wrong documents with rive get high scores. Result: top list holds river banks, money-bank answers stay hidden. Sense-check: no re-ranking of the wrong list can bring back banque pages, since they never entered the candidate set.

15.3.3 Query Document and Pivot Paths Compared

Query translation (mapping the query into the document language), for example English terms mapped word by word into French before French documents are retrieved, is direct and easy to explain. Queries in English map word by word into French, for example, and French documents are retrieved. It is flexible because nothing in the stored corpus is touched. New queries arrive and map on the fly. The store never rebuilds.

The weakness is context. Three or four words give almost no context. A full paragraph gives context. Two or three words do not. Results can stay ambiguous. A familiar monolingual hint of the same pain is the "did you mean" prompt in web search. Even in English, a mistyped query triggers a suggestion and results follow the suggestion. With cross-lingual queries the ambiguity is larger, because each short word may hold several foreign forms and there is no surrounding text to pick among them.

Document translation (mapping documents into the query language), for example rendering French pages into English once and then running English queries, keeps context. A document has at least a paragraph, often 500 to 600 words, so sense is easier to fix. It is strong against noise for that reason. Longer text lets the translator see which sense fits. The weakness is scale. Each document is large. Translating many documents costs a lot in compute and in storage. A 500-word file costs about 100 times the work of a 5-word query, and that factor repeats for every file in the store.

Interlingua or pivot translation (mapping both query and document into a third shared language), most often with English as the pivot, is the middle path. The third language is most often English. Suppose the query is in Malayalam and the documents are in Tamil. Both map into English and match there. This helps when direct resources between the pair are missing, for example when no Malayalam documents exist for a query and the path must pivot through English. The cost is high because two translations run instead of one. Performance stays low relative to direct paths, and the amplifying effect matters because document translation was already costly in storage and compute. Errors also chain: a Malayalam-to-English slip plus an English-to-Tamil slip can stack. Use it when direct translation lacks resources. Otherwise prefer direct query or direct document translation.

Picture three pipes feeding one ranker. Pipe one is thin and fast (query path). Pipe two is wide and slow (document path). Pipe three is a V shape through a middle tank (pivot through English). The x-axis is cost, the y-axis is context per token. Query sits low-cost, low-context. Document sits high-cost, high-context. Pivot sits highest-cost, middle quality. One-sentence takeaway: no pipe wins on both axes, so the collection decides.

Real-world placement: web search largely uses query translation because the index is huge and fast-changing. Patent, legal, and product-manual stores use offline document translation because stability lets one translation serve many queries. Malayalam-to-Tamil requests with no direct resources pivot through English.

15.4 Machine Translation Approach

15.4.1 Query Translation Block and Running Example

How does a ready-made translator plug into search?

In the machine translation approach (automatic conversion of query or document by a machine translation component), written as a box $MT$ placed before ranking, query translation works as follows. The query in language one, written $L1$, must become language two, written $L2$. Documents stay in $L2$. A translation module sits before the retrieval model. The retrieval model then does its normal job and returns ranked documents.

Take the query "best food" or "where can I find best restaurant" in Rome. The query is in English. It must map into Italian. Retrieval then runs over an Italian corpus and returns matching documents. Forms such as ristorante and migliore now match. Only the query was translated. The whole corpus stayed untouched. That is why this path is cheaper: about 5 tokens move per request instead of millions.

Strengths are that it is straightforward, easy to explain, and relatively fast because only the query moves. The weakness is missing context, so retrieval can miss even when the machinery is correct. A 3-word query cannot tell the translator which sense fits, so the top-1 translation may be the wrong one.

Three concrete failure shapes appear with Chinese to English as the running pair. First, some words have no translation at all. The Indic word Jugaad was given as the same kind of case — local vocabulary that people use in queries and documents but for which no English equivalent exists, described verbally as "there is no translation at all". Jugaad names a locally understood fix or workaround with no single English word that carries the same load, so any English rendering either drops meaning or needs a phrase. Second, segmentation splits can map two words to one word or one word to several words. The stated example is "word one and word two in Chinese mapped to one word; if taken as two words the glosses are probe and survey, if taken as one word the gloss is take samples". The choice changes meaning in a large way, so the decision of one versus two units matters: probe plus survey suggests two acts, while take samples suggests one sampling act, and ranking follows whichever reading the translator picks. Third, literal transfer of idioms breaks. The stated example is the English cheer "break a leg"; a literal translation into another language misses the intended cheer and retrieval drifts toward injury or fracture pages instead of stage wishes.

Fixes point toward learning probabilities and checking cohesion statistics, then keeping the candidate with the highest probability. Plain query translation without those fixes keeps these issues, because a fixed top-1 rule cannot see context or frequency.

15.4.2 Document Translation Block and Its Fragility

Document translation mirrors query translation. Target documents are rendered into the query language before anyone searches. That work happens offline, not while the query is being processed. Someone builds the whole corpus in the other language in advance. Retrieval then works on that prepared set. Per-query time stays fast because the heavy work already finished.

Suppose the build assumed English to French and queries are always in French. At query time the run is monolingual only. That is efficient at serve time. The fragility shows when the query language changes. If the need shifts from French to Italian, which is natural for European languages, the prepared French corpus no longer fits. The team must re-translate, re-index, and re-crawl from the start. Everything behind the screen repeats. A store built for one $L_q$ cannot serve a new $L_q$ without a full rebuild.

The same shape in block form is a query plus a document corpus that is converted before the retrieval system, with the rest unchanged. A change in query language breaks the chain into pieces, because the converted corpus, the index, and the cached norms all assume the old target language.

Strengths are that the path is straightforward, the user directly uses retrieved documents, documents carry more context, and behavior is strong against noise compared with query translation. A 550-word page lets the translator resolve sense from neighbors. Weaknesses are the load of translating whole documents plus storage and compute complexity. Both the original and the rendered copy must be stored and indexed, and every update re-runs the pipe.

Picture a timeline with two lanes. Lane one (query path) shows a short spike at query time. Lane two (document path) shows a long offline block before time zero, then a flat fast lane at serve time, then a full repeat block the moment the query language shifts from French to Italian. One-sentence takeaway: pay once and serve fast, but pay again in full when the language need moves.

15.4.3 Practical Choice Rules

The choice rule taught here is compact:

• Small, stable, high-value collection favors document translation. Legal retrieval, patent retrieval, and product manuals are the named cases. Each file repays its one-time cost across many future queries.
• Huge, fast-changing collection favors query translation. Web pages churn too fast for offline builds to stay fresh, so per-query mapping wins even with thinner context.

A number sketch fixes the point. Let English query vector be $q=[0.9,0.1]$ and French document vector be $d=[0.9,0.1]$ in their own spaces. Their cosine is 1.0, yet the match is empty if axis one means past-tense in one space and plural in the other. Same numbers, different truths. Only a shared space lets the dot product mean agreement.

Real-world placement: product manuals and legal collections use offline document translation because stability lets one translation serve many queries. Web search uses per-query mapping because the index never sits still. The Rome restaurant run (English query mapped into Italian over an Italian corpus) is the canonical query-side trace to remember.

15.5 Dictionary Translation and Cohesion

15.5.1 Word by Word Mapping With a Bilingual Dictionary

What if no full translator exists, only a word book?

In dictionary-based translation (word by word conversion using a bilingual dictionary), where each source term $s$ maps to a set $T(s)$ of target forms, someone holds a dictionary that gives word to word meanings. The idea is simple and intuitive. No grammar engine runs. Each query word is replaced by a dictionary form, and retrieval runs on the replaced query.

Take the English query "data access" to be rendered into French. The dictionary gives four different French forms for access and three different forms for data. The stated shape is "access alone has four forms, data has three forms". The question is which two words to pick as the translated query. Reference word books show the same spread: data maps to forms such as donnee, materiau, and data, while access maps to forms such as attaque, acceder, intelligence, entree, and acces. Spelling closeness misleads here: a form that looks like access need not mean access in the query sense.

Three selection styles were compared. First, pick the first dictionary entry for each word. That is the lay default. The hope is that the first entry is the most frequent translation. Accuracy in that style was stated as around 70 percent in the discussion, described verbally as "maybe 70 percent accuracy if I go for first word alone". The risk is a wrong pick and a missed synonym. For access, a later form may look closer in spelling to access and may read like the better pick, yet it is not first. First-entry also ignores the other query word, so data cannot rescue a bad access pick.

Second, build all combinations. With four forms for one word and three for the other, that is $4\times 3=12$ combinations, stated verbally as "I create all 12 combinations". Run the query 12 times and one combination should succeed. The cost is many incorrect translations. Those extras add load and add noise. With 50 translations per word the count explodes ($50\times 50=2,500$ runs for a 2-word query), so brute force does not scale. Noise also hurts ranking, because 11 wrong queries pull wrong documents into the merged list.

Third, use cohesion. That is the smarter choice among the three. It keeps one run but picks the pair that fits together in the target language.

15.5.2 Cohesion Frequencies and Smartphone Analogy

Cohesion (the tendency of the right pair to appear together often in the target language), measured as joint frequency in target text, is counted for each candidate pair as how often those two words appear together in French. Collect their frequencies. Keep the combination with the highest frequency and use it as the query.

The kept steps are "for each pair, count how often these two words appear together in French" and "whichever combination gives the highest frequency, take it for the query". Practical systems handle the leftover noise by weighting each word instead of keeping all forms equally, so near-ties do not force a hard single pick.

Think of a smartphone keyboard. It predicts the next word not by deep meaning but by cohesion frequencies stored in the back end. After you type data-like text, it suggests the access-like word that most often followed in past typing. Dictionary cohesion works the same way: past joint use predicts the right current pair. Where the analogy breaks: a keyboard learns your personal style, while cross-lingual cohesion learns from a large target corpus, so domain shift (chat text versus legal French) can tilt counts the wrong way.

Formally, let the source query have terms $s_1$ and $s_2$. Let candidate target sets be $T(s_1)$ and $T(s_2)$. The cohesion score for a pair $(t_1,t_2)$ with $t_1\in T(s_1)$ and $t_2\in T(s_2)$ is the co-occurrence count $c(t_1,t_2)$ in target text, where $c$ counts joint appearances in the same text span. The chosen pair maximizes that count. Reference treatments refine the same idea with similarity scores such as pointwise mutual information, Dice, or mutual information summed over the query, and pick the set with the largest total. The lecture form stays with raw joint frequency, which matches those refinements in spirit: keep the pair that appears together most.

15.5.3 Mathematical Formulation

Let source query $Q_S$ hold source terms. Let bilingual entries map each source term $s$ to a set $T(s)$. Let target corpus counts give $c(t_1,t_2)$ for joint appearance. Then the translated query $Q_T$ is the pair that maximizes $c$.

The selection rule in block form is:

$$Q_T=\arg \underset{t_1\in T(s_1),t_2\in T(s_2)}{\max }c(t_1,t_2)$$

The verbal audit kept alongside is "whichever combination gives the highest frequency, take it for the query". Here the argmax runs over $3\times 4=12$ pairs. Each pair contributes one number $c$. The max wins. A tiny instance: if $c=1,240$ for one pair and below 310 for all others, the 1,240 pair is $Q_T$.

Picture a 3-by-4 grid. Rows are data forms, columns are access forms. Each cell holds its joint count. One cell glows at 1,240 while the rest stay dark below 310. The x-axis is access forms, the y-axis is data forms, cell brightness is count. One-sentence takeaway: cohesion turns a flat dictionary list into a lit map where the brightest cell is the query.

15.6 Parallel Text IBM Model 1 Alignment and Expectation Maximization

15.6.1 What Parallel Text Means

Where do translators learn if no word book is enough?

Parallel text (the same content available in two languages), stated as "texts with their translations in another language", underlies the third approach. At first that sounds odd. Why would the same document exist in French and in English?

Because it often does. An airline safety card shows safety steps in one language and the same steps in another language on the same printed card, with images repeating. An exam paper shows English on the left and Hindi on the right. A product guide shows the same guide in two languages. The web holds the same pattern. A Korean college page shows Azad University in English and its Korean twin alongside, so parallel pages exist on the web. The basic assumption is that the same content is available in another language. Word to word mapping may be absent, but topic-level translation is present. Parallel text implicitly holds translation even when it is not word aligned: the sentence pairs are known, the word links inside them are not, and learning fills those links.

The first technique in this family uses retrieval itself as the translation engine. Take a French query. Retrieve French documents from a parallel corpus. Take their parallel English twins. Pull keywords from those twins. Build an English query from those keywords. Retrieve English documents with that query. Retrieval runs twice. The rest matches earlier translation thinking, except no machine translation engine and no dictionary is used. The hope is that the twins hold known translation words. Without a parallel corpus those tokens are only noise, because twin links are the only bridge.

15.6.2 Translation Probabilities and Bayes View

From twins to numbers: how likely is each word sense?

Training estimates the probability of a target word given a source word, written $P(t\mid s)$, where $t$ is a target word and $s$ is a source word. The kept description is "estimate the probability of target word for the given source word". The more often a French word shows up in passages paired with a given English need, the higher that probability should be. Counts across twins become weights.

Bayes view (flipping conditional probabilities through prior and likelihood), used here to favor likely readings with soft weights, supports the soft choice. Where hard 0 or 1 membership fails, probabilities give a soft value of belonging to one reading with some weight and to another reading with another weight. IBM Models 1 to 5 all use such translations with different details. Model 1 is the focus here: it keeps only lexical weights plus uniform alignment, which is enough to learn useful $P(t\mid s)$ tables.

Two lexical examples show why probabilities beat possibilities. The German word Haus can map to house, building, home, household, and shell, where shell is the shell of a tortoise or a snail seen as its house. The candidates are not equally likely. House deserves more weight than building. We call building less often than house. With lexical knowledge we can safely give house the larger probability. A concrete table makes it real: $P(\text{house}\mid \text{Haus})=0.60$, $P(\text{building}\mid \text{Haus})=0.15$, $P(\text{home}\mid \text{Haus})=0.12$, $P(\text{household}\mid \text{Haus})=0.08$, $P(\text{shell}\mid \text{Haus})=0.05$. The five sum to 1.00. Likewise spring can mean season, metal coil, or water source. The moment we hear spring, season feels most likely. Likelihood of season for spring is largest, say $P(\text{season}\mid \text{spring})=0.55$ against $P(\text{coil}\mid \text{spring})=0.25$ and $P(\text{source}\mid \text{spring})=0.20$. So translation keeps likelihoods, not bare lists.

An English modeling detail for synonyms: the true query idea may be concession, while official documents write fare or rebate. Terms such as rail fare rebate and rail fare concession should both retrieve. Training must not discard those variants. That is where the complexity enters: the table must spread weight across true synonyms instead of collapsing to one surface form, or recall drops on official wording.

Picture a set of bars per source word. The x-axis lists target readings (house, building, shell). The y-axis is probability. House towers at 0.60 while shell sits near 0.05. One-sentence takeaway: twins vote with counts, and the tallest bar becomes the query-time pick while shorter bars stay as backups.

15.6.3 Alignment Positions and Reordering

Words move. How do links track shifting slots?

Alignment (the link from a source position to a target position), written as a map $a$ from slots to slots, is learned alongside lexical weights. Let source positions be indexed by $i$ and target positions by $j$. Then alignment $a$ maps $i$ to $j$, written $a(i)=j$. The kept description is "alignment of I, which is from source, to J, which is target". A null slot, written $0$, lets an English word arise from no foreign word, which handles insertions such as articles with no counterpart.

A tidy one-to-one case helps. Take the German sentence Das Haus ist klein rendered as the house is small. The stated gloss is "DAS can be the, house can be house, ist can be is, klein can be small". Here first maps to first, second to second, and so on. That is a clean translation with $a(1)=1$, $a(2)=2$, $a(3)=3$, $a(4)=4$.

That tidiness often fails. Reordering happens. In English we say "I am having a meal" with meal last. In Hindi the word for food, Khana, need not be last. Order crosses. One word can map to two words. A word can have no counterpart. An extra word can appear on one side only. The French to English pair with red mapping to rouge and car mapping to voiture shows the same point — even when the lexical links are right, positions do not line up one to one and word order shifts, since French places the adjective after the noun while English places it before. Reference treatments name the same effects with textbook pairs: solar system rendered with reversed order in French, and one source word mapping to several target words through fertility. The lecture keeps the French illustration qualitative (exact string left as heard), and the mechanism matches the references: distortion moves slots, fertility splits or drops them.

Once alignment is known, it feeds probability estimation for the sentence pair. Each English slot $j$ draws its word from the foreign slot $a(j)$, so the sentence score multiplies one lexical pick per slot, weighted by how many alignments were possible.

15.6.4 IBM Model 1 Equation and Worked Maximization

Now the full scoring rule with all symbols named.

The final Model 1 form taught here has a normalization term and a lexical product. Let $f$ be the foreign sentence with length $l_f$. Let $e$ be the English sentence with length $l_e$. Let $ϵ$ be the normalization constant, a fixed length-model weight with $ϵ>0$. Let $t(e_j\mid f_i)$ be the lexical translation probability of English word $e_j$ given foreign word $f_i$. The kept description is "first term is normalization, in the denominator length of foreign plus one to the power length of English, then likelihood or translation probability, product over all words".

Why the plus one: each English position may link to any of the $l_f$ foreign words or to null. That is $l_f+1$ options per slot. With $l_e$ independent slots in Model 1, the count multiplies to $(l_f+1{)}^{l_e}$. Dividing by it keeps the distribution normalized over alignments. The $ϵ$ factor holds the length-model weight so that different English lengths stay comparable.

A German worked case shows use. The foreign length is 4 and the English length is 4, so the denominator is $(4+1{)}^4=5^4$, described verbally as "five to the power four". Lexical tables are assumed known. For example, the German function word in first slot has possible English readings the, that, which, who, and this, each with a lexical weight. For Haus the top weight favors house over building and shell. For ist the top weight favors is. For klein the top weight favors small or little.

Concretely, take the highest lexical pick for each slot and multiply. That product times the normalization constant gives the sentence score. The best sentence is the house is small in the discussion, because those picks hold the largest values. A check pair such as the shell has petty multiplies smaller entries and scores lower. Simple products decide.

15.6.5 Expectation Maximization From Uniform Starts

Who teaches the tables when no word links are marked? The loop teaches itself.

Model 1 faces a chicken-and-egg loop. If complete word alignments were marked, estimation would be trivial: count links and divide. If model probabilities were already known, gaps could be filled: link each word to its best mate. Neither holds, so expectation maximization (an iterative method that fixes one set of parameters while tuning the other, then repeats to convergence), used as in k-means and Gaussian mixture models, is the tool. Start with uniform values in the first pass. Fix one parameter and optimize the other. Repeat until convergence. Final values give posterior probabilities for target words given source words.

Purpose: learn $t(f\mid e)$ tables from sentence twins alone, with no hand-marked word links. Inputs: a set of aligned sentence pairs plus a vocabulary on each side. Outputs: lexical tables $t$ that maximize twin likelihood. Steps: (1) set all $t$ uniform; (2) expectation: spread each observed co-occurrence across possible links in proportion to current $t$; (3) maximization: re-estimate $t$ from those fractional counts by dividing joint counts by marginal counts; (4) repeat steps 2-3 until shifts fall below a small threshold. Rationale per step: uniform start avoids early bias, expectation fills the missing links softly, maximization turns soft fills into better tables, and looping lets the two improve together.

A three-phrase French to English toy shows the intuition. Start uniform so each foreign word is equally likely to map to each English word. For example, la is equally likely to map to the and to other mates, and house is equally likely to map to la and to maison. That is the expectation start: every line has equal width.

Then counts speak. In the three phrases, la appears three times. It meets the sense three times, house sense two times, and other senses fewer. The model learns that live alignment to the is common. Blue meets blue once. Flower meets flower once. La meets house twice, blue once, and flower once. Capture that signal by line width. The line between la and the grows thickest. The line to house is next. Lines to flower and blue stay thin. Repeat for other words. Fleur has only the flower link. Maison meets house twice. Blue meets blue once. When no alignment updates change further, the run has converged. There is no single straight one-to-one map. Cross links such as bleu to blue and maison to house remain, with widths set by final $t$. Those learned weights are the lexical probabilities used later to maximize the sentence score.

A practical init example: if Haus has two readings, house and building, start each at 0.5. So $t(\text{house}\mid \text{Haus})=0.5$ and $t(\text{building}\mid \text{Haus})=0.5$. Then assign missing alignments in proportion to 0.5/0.5, re-estimate from completed data (suppose counts move the split to 0.75/0.25), and repeat. That is the same fix-one, tune-other loop in words. After a few rounds the split might read 0.85/0.15 and then stop moving, which signals convergence.

Posterior language was also reviewed. Posterior is likelihood times prior divided by evidence $P(S)$, where evidence acts as a normalization factor. Most optimization work keeps the numerator only. The kept form is "posterior is likelihood times prior; denominator is evidence and acts like normalization; numerator alone is enough". In symbols with $E$ for English and $F$ for foreign, one inline form is $P(E\mid F)$ for posterior, with $P(F\mid E)$ for likelihood and $P(E)$ for prior. The maximization keeps the setting with the largest likelihood-prior product:

$$P(E\mid F)\propto P(F\mid E)P(E)$$

Here $P(F\mid E)$ comes from the Model 1 product above, $P(E)$ is a language-model prior that favors fluent English, and the missing denominator $P(F)$ is shared across candidates so it drops out of the max. Complexity per pass scales with total twin tokens times vocabulary fan-out, which is why tiny toys converge by hand while real runs need indexed counts. Use this trainer when sentence twins exist but word links do not; prefer a dictionary or direct translator when twins are missing.

15.7 Comparable Corpora and Multilingual Merging

15.7.1 Limits of Parallel Data and Web Mining

Strong medicine, short supply.

Parallel corpora have a hard limit. They do not exist for all documents. Rare and minor languages are barely covered at all. That is the core weakness of an otherwise strong model. Large curated twins exist for a few European pairs, but many other pairs have almost nothing curated, and building twins by hand costs far too much per sentence.

Two web-mining ideas were introduced to find more parallel text. One uses pages that link to each other across languages and reads language cues in anchors. That method was named STRAND in the discussion. STRAND starts from a home page that points to an English and a French version with anchor texts such as English and French, then checks that the two targets share structure before accepting them as twins. The other uses URL patterns that signal language versions, such as paired names like report_en.htm and report_fr.htm or path tags like /en/ and /fr/. Both try to extract parallel text where none was curated. Both rely on the habit that bilingual sites mirror layout across languages.

15.7.2 Comparable Corpora and Weak Signals

What if twins never arrive but cousins abound?

Comparable corpora (texts in two languages on the same topic without strict sentence-to-sentence parallelism), for example a French page and an English page that each explain cross-lingual retrieval without matching line by line, relax the need. Texts in two languages concern the same topic without being strict parallels. A French page on cross-lingual retrieval and an English page on the same topic may both explain the same ideas without matching line by line. Newswire articles on the same event behave the same way: same bombing, same match, same summit, but written apart with different sentences and different details.

The signal then is co-occurrence across the collection. If an English term and a French term keep appearing in documents that cover the same events, treat that cohesion as translation evidence. Much more data is available in this mode, but the alignment signal is much weaker. Weak signals bring noise. Noisy queries retrieve noisy outputs because no sentence points and says this is the translation of that. Counts across topic-matched documents replace counts across sentence twins, so accidental co-occurrence (two unrelated words that share big stories) pollutes the table. Even so, this path is more practical because strict parallels are rare. Cousins are everywhere; twins are not.

A number contrast fixes the trade. Suppose twins give 10,000 sentence pairs with 90 percent clean links. Cousins give 1,000,000 topic pairs with 20 percent clean links. Twins yield about 9,000 good links. Cousins yield about 200,000 good links mixed with 800,000 noisy ones. Takeaway: cousins win on volume, twins win on precision, so cousin methods need heavier filtering before their tables can be trusted.

15.7.3 Multilingual Merging

One query, many languages, one list. What is new?

Multilingual retrieval (cross-lingual retrieval plus merging across languages), run as retrieval per language followed by one merge, runs retrieval in each language and merges. The genuine problem is not retrieval but merging. How should documents from each language be ranked together and matched into one list? Scores from an English index, a French index, and a Hindi index live on different scales (different idf, different lengths, different translators), so raw scores cannot be sorted together. Once cross-lingual steps are known, multilingual work is a superset that adds that merge: normalize per-language scores or map into shared features, then interleave.

Picture three sorted cups (English, French, Hindi) poured into one jug. Each cup has its own scale marks. Pouring raw scores mixes scales. A merge rule re-marks all cups to one scale first, then pours. One-sentence takeaway: match per language, then calibrate, then merge.

15.8 Multimedia Retrieval Scope Challenges and Search Methods

15.8.1 From Text Only to Many Media

What happens when the answer is not words at all?

So far retrieval considered only text. Information retrieval covers representation, storage, and retrieval of unstructured data, which is not text only. Image, sound, video, and other media databases need retrieval too. Objects in those stores do not arrive as words, yet retrieval is still needed. Scaling text ideas to other unstructured forms gives multimedia retrieval: same rank-and-return loop, new features per medium.

Multimedia (one or more interlinked digital media such as audio, video, picture, or volumetric data), for example a class session with diagrams plus speech plus screen activity plus annotations used together, appears in simple combos. A class session with diagrams plus speech plus screen activity plus annotations is multimedia. Using all of that evidence together makes retrieval stronger than video alone, because speech names the slide, the slide draws the shape, and the annotation marks the key step.

A multimedia query (a request expressed in media form), such as pixels for image search or a hum for audio search, varies by type. An image query can be pixel information. An audio query can be a time-varying signal. A video query can be sound plus time plus frames. Queries name the same need in different signal forms: show me more like this picture, find this tune, fetch clips shaped like this motion.

What the system looks for depends on internal structure. Attributes, text, two-dimensional gray levels, one-dimensional time series, digitized voice, fingerprints, and videos each have their own structure. The system captures that structure as extracted features and compares those features with the query. In text work the main feature is text. In multimedia work the answer rarely comes from one feature. A video run may combine frame cues, sound fingerprints, and temporal cues as joint evidence. A three-dimensional picture element is called a voxel (the 3D counterpart of a pixel), a value at position $(x,y,z)$, while a two-dimensional picture element is called a pixel (the 2D unit), a value at $(x,y)$. Use voxel info for 3D and pixel info for 2D. A gray pixel holds one number 0 to 255. A color pixel holds a triple $(r,g,b)$. A voice frame holds spectral slices over time. Each form needs its own feature extraction step before any comparison.

Real-world anchors: medical scans where tumor-like structure in any view should trigger return, so a sagittal query can pull a coronal twin. Market series where similar price movement should group together, so two firms with matched dips sit side by side for the invest-or-not call.

15.8.2 Core Challenges

Four challenges recur. The semantic gap (the distance between low-level measures such as pixels or frequencies and high-level ideas), for example pixels reading orange against the idea celebrating, is first. A machine can read color, edges, and spatial pattern. At most it may report a distribution of values. It cannot on its own report that people in a photo look happy and are celebrating. That extra meaning needs bridging across low-level features, objects, and context. Pixels say orange and round. People say sunset joy. The gap between those two readings is the core research problem.

Polysemy (the same word or signal supporting different readings by context), for example one curve meaning rally in one market and noise in another, is second. One signal can mean several things. Context must disambiguate. A round orange patch is a sun in a sky photo and a lamp in a room photo. Features alone cannot pick; surrounding cues must vote.

The fusion problem (how to combine evidence from text, image, audio, and video), for example weighting frame cues at 0.5 plus sound at 0.3 plus text at 0.2, is third. Each medium gives a partial view. The merge must weight them well. Bad weights let one noisy channel outvote two good ones.

Responsiveness (returning answers fast even though many media are processed), for example under 2 seconds on a million-image store, is fourth. A user will not wait ten minutes because the back end touched many media stores. High-end compute can help, but it is costly. A useful system balances all four and stays quick, often with indexed features and short-list re-rank instead of full scans.

Picture a gap diagram. The x-axis runs from pixels to objects to meaning. The y-axis is machine confidence. Confidence starts high at pixels (color counts are easy), dips at objects (which blob is the sun?), and falls further at meaning (are they celebrating?). One-sentence takeaway: features live on the left, user needs live on the right, and every method in this lecture is a bridge rightward.

15.8.3 Metadata Piggyback Automated and Fingerprint Search

Four search technologies were covered. Each answers a different question about the same store.

Metadata retrieval (search using information about the object), such as format, owner, or location fields, uses data that is not strictly needed to render the object but helps retrieval. Technical metadata (format, dimensions, duration, and similar facts), for example JPEG plus 1920 by 1080 plus 42 seconds, plus administrative metadata (ownership, rights, location, and similar facts), for example owner plus license plus archive shelf, add richer context alongside actual content.

The hard questions are who vouches for accuracy, whether the entry is correct and consistent, whether scientific detail follows a shared form, and whether all documents even carry metadata. A simple field can mislead. The Dublin Core creator field, written $dc:creator$, can hold an organization name, a person name, or a shortened form. Without a shared vocabulary the same field means different things: one archive writes a lab name, another writes a technician, a third writes initials. Broad metadata needs agreed vocabularies and genuine entries. Manual entry by an authorized person gives a better version. Automatic generation can also work when strong rules guide it, such as file-header parsers for technical fields.

Piggyback retrieval (text-query search over text attached to media), for example subtitles standing in for video bytes, avoids analyzing media bytes directly. For video use subtitles. For songs use lyrics. For images use captions or surrounding art notes. For documents use titles, subtitles, descriptions, and nearby text. Match the text query with standard document methods, then return the linked media objects. It is efficient and easy to explain. It works only when attached text truly describes the object. A wrong caption still retrieves, but the object is not the one wanted, so quality drops. A travel video tagged beach by error returns for beach queries while showing a parking lot.

Automated annotation (generating descriptive labels from media signals), for example mapping regions to water, grass, trees, and sun, learns the bridge. Visual regions map to labels such as water, grass, trees, and sun. A sample Bayesian classifier can align image regions to text labels of form $P(\text{label}\mid \text{region})$. The model learns statistical links from labeled examples. Predicted labels then connect query to retrieval. Accuracy leans on training data, chosen features, and how well the model handles ambiguity and many objects in one scene. Small training sets and crowded scenes pull scores down fast.

Fingerprinting (matching by compact signatures), for example a hash that survives resize or compression, answers a different question: is this the same item or a transformed copy. The query here is not text but media. A small region showing the Leaning Tower of Pisa can match the full image by signature. The hope is stability under resize, compression, added Gaussian noise, or cropping. A cropped or noisy copy should still match. Usual uses are duplicate detection, copyright monitoring, and legal matching. For audio, a spectrogram signature matches songs or voices to stored media: a 10-second hum finds the full track even after re-encoding.

Real-world placement: duplicate detection and copyright monitoring with image and audio signatures, including spectrogram matching for songs and voices. Subtitle search for video, lyric search for songs, and caption search for images cover the piggyback lane. Dublin Core fields cover the metadata lane where vocabularies are shared.

15.9 Content-Based Image Retrieval and Histogram Comparison

15.9.1 Feature Vectors and Similarity

How does a picture become the query?

Content-based retrieval (search where media content queries media stores), with image-to-image as the running case, works as follows. The user brings multimedia content. The system checks it against an image store and returns valid matches. The system is not searching for a keyword such as tree. It searches for patterns that resemble the example under the chosen feature representation. Same-medium matching is the contract here: image matches image by shared visual features.

Each image, query and stored alike, goes through the same steps: pre-process, extract features, then compare. Features include color, texture, shape, and related visual cues as a vector. Pre-processing evens size and light. Extraction turns the cleaned pixels into a short number list. Comparison scores those lists.

A worked cosine trace with real numbers fixes the reading. Let $q=[3,4]$ and $d=[6,8]$. Dot: $3\times 6+4\times 8=18+32=50$. Lengths: $\Vert q\Vert =5$, $\Vert d\Vert =10$. Score: $50/(5\times 10)=1.00$. Same direction gives 1.00 even though brightness doubled. Now let $d_2=[4,-3]$. Dot: $3\times 4+4\times (-3)=12-12=0$. Score 0.00 for a right-angle mismatch. Rule: 1.00 means same direction, 0.00 means unrelated direction, and the max over the store wins.

Scores must be read against representation, not against brightness, viewpoint, or background. An apple-to-apple pair with different lighting and different viewpoint still scored 0.91 in the example because learned features lined up. That high value is meaningful only for that representation: shape stayed round, warm color stayed warm, smooth region stayed smooth, centered composition stayed centered. An apple-to-orange style pair scored 0.78, lower than apple-to-apple but still fairly high. That makes sense if the representation captures round shape, warm color, smooth region, and centered composition, since both items share those cues. If color weight grew, that score would fall, because hue would then split them. A pair with nothing shared in those cues scores far lower, near 0.10 to 0.20. The crux lies in building good vectors. Good extraction lowers the pull of artifacts such as shadows and cluttered backs.

A foliage query that also returned an atomic mushroom cloud shape was noted as a pattern-level match, since shape cues overlapped even though meaning differed: layered lobes above a stem read like a cap above a stalk at the feature level. Current tuning has improved such cases by adding texture and context weights, but the example shows why representation choice matters. Shape alone confuses trees with blasts. Shape plus texture plus context separates them.

Picture a 2D arrow plot. Query arrow points northeast. Stored arrows fan around it. The x-axis is feature one, the y-axis is feature two. Angle from the query arrow sets the score: small angle near 0.91, wide angle near 0.20. One-sentence takeaway: cosine ranks by angle, not by length, so twins at different brightness still meet.

15.9.2 Histogram Worked Comparison

A color histogram (the frequency distribution of red, green, and blue values), counted per channel over 0 to 255, gives a simple global feature. For a query image the distribution might read red high, green medium, blue low. Build the same three-channel histogram for each of five stored images. Compare query histogram to each stored histogram. The most similar histograms return.

Pixel-to-histogram steps were shown for gray and color. In gray levels, 0 is black and 255 is white. Count how often each level appears in the patch and plot frequency against level. For a tiny patch the counts form the histogram directly: a 4-pixel patch with levels $[0,0,255,128]$ gives counts black 2, mid 1, white 1. In color, the same count runs per channel. Black sits at $(0,0,0)$ for red, green, blue. White sits at $(255,255,255)$. Other shades fill the voxel space between. The kept anchor is "zero for black, 255 for white; black at zeros corner, white at 255 corner". Gray needs one axis. Color needs three, one per channel, stacked or side by side.

Histograms alone can mislead because they keep only color. A sunset query matched against another image with the same color balance can score high even when content differs: two frames with red-high green-medium blue-low overlap strongly as histograms while showing different scenes (beach sun versus city glow). The kept warning is "histogram comparison alone is not sufficient because it only looks at colors". Color-blind viewing was mentioned as a reminder that color perception itself varies, which reinforces why color-only features are brittle: a representation that leans on one channel family breaks for viewers and cameras that weight channels differently.

15.9.3 Student Questions and Answers

A third doubt often resurfaces here, so it is closed once. With vectors holding context, why not embed any language directly and compare numbers? Because vectors from different language spaces do not share axes. Conversion to shared ground must come first, by translation or by joint training. Only then does similarity mean anything. An apple pair at 0.91 works because both arrows live in one visual space. A Tamil query arrow and a Kannada document arrow live in two spaces until a bridge joins them.

Exam Guidance Summary

• Expect a small numerical on translation choice. Given lexical tables, multiply candidate probabilities and keep the max, as in the German four-word trace where the house is small beats the shell has petty. No full expectation maximization iteration will be asked in a pen-and-paper setting. Revise product-then-max, plus the denominator rule $(l_f+1{)}^{l_e}$ with the $5^4$ instance.
• Exam note: for "small, stable, high-value" such as legal, patent, or manuals, answer document translation. For "huge, fast-changing", answer query translation. State cost reasons: offline build plus storage versus per-query ambiguity. Use the patent-versus-web contrast as the one-line reason.
• Exam note: for dictionary choice with several forms per word, answer cohesion by joint frequency, not first-entry and not brute-force all pairs. Name the data-access run: 12 pairs, counts looked up in French, top pair kept. Mention weighting in practical systems as the softening for near-ties.
• Exam note: for parallel versus comparable, state the trade. Parallel gives strong alignment but is rare, especially for minor languages, with STRAND anchor cues and URL cues as the mining fix. Comparable gives more data but weak, noisy signals, since no sentence points at its twin. Multilingual adds merging across per-language lists.
• Exam note: for histogram-only retrieval, state the limit. It sees color only and can confuse items with matched color balance, such as sunset-like pairs. Good features must add shape, texture, and composition. Cite the apple 0.91 versus apple-orange 0.78 gap as the representation effect.
• Exam note: use the two weeks before the next session to prepare well and keep prior material fresh. The next session is set to finish multimedia retrieval and to start neural retrieval with joint spaces such as CLIP and ALIGN. Questions can be posted in the team chat for reply during preparation.

Key Industry Applications

• Web search that translates short queries and shows "did you mean" style suggestions when context is thin, since 3-word queries carry little sense on their own.
• Patent search across Korean and English where titles and approvals stay in Korean, with offline renders paying off on a small stable high-value store.
• Crisis field reports in local languages retrieved by English queries when time matters, where a rough fast match beats a perfect slow one.
• Legal pre-selection that cuts 10,000 documents to 10 for human translation, turning a 25,000,000 rupee style bill into a 25,000 style bill at a factor near 1,000.
• Hotel and travel reviews in Italian retrieved for English queries before booking, where the most honest notes sit in the local language.
• Health guidance in state languages reached through cross-lingual queries, from Karnataka, Delhi, and Andhra Pradesh notices to dose and clinic facts.
• Captioned movies where native audio plus captions bridge languages, such as a Malayalam crime thriller watched with caption support.
• Airline safety cards, bilingual exam papers, product guides, and parallel web pages as sources of parallel text, including Azad University style twins mined by STRAND anchor cues and URL patterns.
• Smartphone keyboards that predict the next word by cohesion frequencies, the same joint-count logic as dictionary cohesion.
• Medical image search across sagittal, coronal, and axial views for tumor-like structure, where shape plus density features survive view change.
• Market series grouped by similar price movement to guide invest-or-not choices, using time-series shape over price traces.
• Duplicate detection, copyright monitoring, and legal matching with image and audio fingerprints, including spectrogram matching for songs and voices under resize, compression, noise, or cropping.
• Joint word-image embedding work such as CLIP and ALIGN for text-to-image search beyond same-media matching, set as the next step after image-to-image cosine and histogram baselines.