Introduction to Natural Language Processing

This first session sets the stage for the whole course. It answers four questions: what NLP is, where it sits inside AI, what a modern NLP pipeline looks like, and how you judge whether an NLP system is any good. It also runs a small live demo of text preprocessing with NLTK and spaCy, and it spends a long time on one uncomfortable truth: human language is deeply ambiguous, and that ambiguity is the reason this field exists.

1.1 What NLP Is and Why It Matters

This section answers the opening question of the course: what exactly is NLP, where does it sit inside artificial intelligence, and why does it deserve a whole course? The answer builds up through a definition, a quick tour of what the course will cover, the students' own reasons for studying the field, and a famous movie dialogue that shows how much language skill humans run without noticing.

1.1.1 Definition and Place in AI

Natural Language Processing (NLP) is the branch of artificial intelligence that helps machines understand, analyze, and generate human language. Take the three verbs seriously, because they split the field into three job families:

• Understand — read a sentence and know what it means: which "bank" is meant in "I sat by the bank" (the river bank, not a financial bank).
• Analyze — pull structure out of text: label each word with its part of speech, find the names of people and companies, or measure the sentiment of a review.
• Generate — produce new language: write a summary, translate a sentence, or answer a question in full sentences.

The goal is simple to say and hard to build: machines should do what humans do when they talk to each other. The layered act of understanding that you perform without effort — context, word senses, topic, intent — is exactly what NLP tries to copy.

Where does NLP fit in computer science? Picture a set of nested boxes. AI is the big umbrella — the whole project of making machines behave intelligently. Inside it sit machine learning (systems that learn from data), deep learning (learning with many-layered neural networks), and NLP itself. NLP borrows from all of them: a Naive Bayes classifier is a machine learning technique you might use for sentiment analysis, and transformers are deep learning techniques you use inside language models. NLP also borrows from linguistics, psycholinguistics, and cognitive science, plus math and statistics. So NLP is not one technology. It is an area — the way networks or cloud computing are areas.

That exchange is worth lingering on, because it is the most common category mistake newcomers make. A language model is one family of techniques inside NLP. An agent is a system that uses language models. NLP is the whole subject that contains both, and NLP itself is one branch of AI. This course is about the whole area, not about one model family.

1.1.2 What This Course Covers

The course is built as a fundamentals course on purpose. Industry professionals consulted when the syllabus was designed made one point again and again: a degree-level program should teach understanding of the algorithms, not just API calls. In an agentic AI world where automated agents do most of the execution, the human's job is judgment — knowing when to apply which technique, which algorithm suits which application, and which model makes sense when you have SLMs, LLMs, nano models, edge AI, and physical AI all competing for your attention. The teaching principle is also deliberately old-school: go from known to unknown, easy to difficult, so nothing arrives above your head.

The topic list, in order:

• Vector semantics — the idea that words become numbers. Machines crunch numbers far better than strings of letters, so you convert a word into a numeric vector. A word like "vector" gets represented by a list of weights, and those weights are learned by neural networks. The details of how the numbers are learned come in later modules (contextual embeddings, GPT-style models), but the concept is this: strings in, numbers out.
• Word embeddings — the learned vector representations themselves, covered in the same block of modules.
• Language modeling — starting with simple n-gram language models and moving up to neural language models and an introduction to large language models. The recap of transformers, attention, and LSTM is promised for later sessions (the class has already met attention in a deep neural networks course).
• Classical NLP methods — part-of-speech tagging and parsing. The instructor makes a deliberate argument for keeping these: even in the agentic era, many applications still need them, and the session returns to that point later.
• Post mid-semester: encoder–decoder models, attention mechanisms, contextual word embeddings.
• Disambiguation — natural language is a live language, with new jargon appearing constantly (Gen Z slang and beyond), so resolving ambiguity is a core task.
• Knowledge graphs — and their close relative, ontology. Nobody uses ontology directly anymore, but a knowledge graph is just a populated ontology, and the concept matters for RAG and especially GraphRAG, which is popular in industry.
• Text summarization — the capstone application where all earlier techniques get applied.

Picture a graph with two of these dimensions as its axes. "vector" and "matrix" land as two points close together in a corner; an unrelated word like "mango" lands far away. The whole field of word embeddings is about choosing those numbers so that closeness on the graph means closeness in meaning. One-sentence takeaway: you trade a word for a point, and then meaning becomes geometry.

Two books anchor the course. Jurafsky and Martin is the main textbook — the pioneer book of the field, used by nearly every college, and it covers roughly 80% of the course content. A free PDF of the updated 2024/2025 edition exists, and an Indian edition is also available; the authors also run Stanford's NLP course. James Allen's classic book is old — it predates much of modern deep learning — but its core concepts still hold, and the fundamentals part of this session draws from it. A third reference book is optional and mostly useful for advanced NLP applications later.

1.1.3 Why Study NLP

The students gave their own reasons at the start, and they map well onto the field's selling points:

There is a deeper argument underneath. We judge a person intelligent largely because they communicate effectively and know a lot about the world — finance, medicine, everything. Communication and knowledge are what language carries. Artificial intelligence, at its core, is the project of making machines intelligent, and language is the biggest lever we have on that project. A machine that can use language well can show, explain, and teach what it knows; a machine that cannot is stuck signalling through buttons and sliders.

One more structural fact supports the choice: text is the largest repository of data we have. Even speech and video usually get converted into words before further processing. A speech recording is language; a video's captions are language; text is language directly. This course focuses on text only — multimodal work belongs to other electives — and text alone is already enormous.

The market backs this up. Growth in LLMs and agentic AI has been explosive, and the job roles are real and named: NLP engineer, prompt engineer, data scientist, conversational-AI-focused data scientist, AI engineer, conversational AI engineer, computational linguist. Companies such as OpenAI and Anthropic became giants on the back of language models — and language modeling is a core NLP concept. In a bank or hospital, the NLP person is the one who knows which of those roles needs which technique. So NLP is hot, well-funded, and sitting at the center of current AI.

1.1.4 The Hal–Dave Test: Reading Language Cues

The session opened its content with a worked example designed to show how much implicit language skill humans have. The class was shown a short two-person dialogue and asked: who is the machine, and who is the human?

The dialogue is from 2001: A Space Odyssey: Dave gives Hal a command, and Hal replies, "I'm sorry, Dave. I'm afraid I can't do that." — and then suggests that Dave take a stress pill and get some rest. The class had to identify the machine. The discussion is worth replaying in full, because every answer is a distinct language cue.

Notice what happened during that discussion: the class used formal register, sentence length, grammatical correctness, politeness, conversational roles, and emotional framing — all at once, without effort. That is the skill NLP wants to build. A related practical observation: generated text has fingerprints. A summary written by ChatGPT tends to differ in style from one by Claude or Gemini. With practice you start to spot the pattern in how each model formulates language.

There is a second layer in the same dialogue. Hal suggests a stress pill and rest. That is an attempt at emotional intelligence — reading Dave's state and responding to it. Human communication also runs on tone: you can hear anger in a voice that goes too quiet or too screamy, you can spot a forced smile from gestures that don't match the words, and you resolve situations by context ("look at the situation" is how parents explain outbursts to kids). Machines today can parse the words but are not yet emotionally intelligent — some are picking it up slowly, but this gap is exactly what makes language processing hard. The machine is only as good as its training data: garbage in, garbage out.

1.1.5 Student Questions and Answers

The confusion keeps coming up, so pin it down one more time: area, part, and component. NLP is an area (a whole subject). A language model is a part of it (one family of techniques). NLP itself is a component of AI. When you hear "LLM", think "one tool on the workbench", not "the workshop".

1.2 A Short History of NLP

Every field looks inevitable in hindsight. This short timeline shows otherwise: NLP reinvented itself every couple of decades, and each reinvention happened because the previous approach hit a wall. Knowing the walls is the fastest way to understand why today's systems look the way they do.

1.2.1 Rule-Based Systems: Eliza and the Turing Test

The field started with rule-based systems, in an era with no compute power and no training data. With no data to learn from, humans wrote the knowledge down by hand — and for narrow domains, the rules worked well.

Two landmarks from that era. The Turing test: Alan Turing's challenge of whether a human can tell if they are talking to a machine or another human. The setup is a typed conversation: a judge exchanges text messages with two hidden partners, one human and one machine. Turing's claim: if the judge cannot reliably say which is which, the machine has won. Notice that this is a behavioral test — nothing about what happens inside the machine. That is also how modern NLP is judged, which is why the test matters.

And Eliza, an early rule-based chatbot — a kind of ancestor of Alexa — built in the 1960s by Joseph Weizenbaum at MIT, originally to converse with depression patients. Eliza had no statistics and no learning at all. It ran on handwritten rules: a script of patterns matched against what the patient typed. Type "I am sad", and a rule rewrites it into the question "Why are you sad?"; type "my mother makes me angry", and it asks "Tell me more about your mother". The trick was that the rules echoed the patient's own words back at them, so the conversation felt understood. Remarkably, patients did find some comfort talking to it — some asked for privacy so they could "speak" to it alone, even though they knew it was a program. This became known as the ELIZA effect: people read human intention into a system that merely reflects their words.

Even geopolitics helped. During the Cold War standoff between the US and Russia, early machine translation ran on rule-based approaches and proved genuinely useful. The 1954 Georgetown–IBM experiment translated a handful of Russian sentences with a hand-built dictionary and grammar rules, and it worked well enough to launch a decade of funding. The point for this course: with a small, well-defined task — sixty sentences about chemistry — rules alone are enough.

1.2.2 Symbolic, Statistical, and ML-Based NLP

Next came symbolic NLP: context-free grammars, knowledge-based systems, and expert systems that encoded grammar and world knowledge by hand. A context-free grammar is a set of rewrite rules like "a sentence is a noun phrase followed by a verb phrase" — writing S → NP VP — with a lexicon that maps categories to words. Knowledge-based systems store facts in a database of logic; expert systems add if-then rules collected from human experts. The weakness was shared: everything had to be typed in by a human, and language kept refusing to fit the boxes.

After that came statistical NLP — data-driven approaches that became possible once data itself became available in quantity. Instead of writing rules about how language should work, you count how language does work in large text collections (corpora), and let probabilities decide: which word usually follows "the", which translation of a word is most common. The same grammar question — "is this sentence structure likely?" — now got a number instead of a yes/no.

Machine learning approaches followed: SVMs, decision trees, and the like. But ML approaches demanded labeled training data and manual feature engineering. Feature engineering is the job of hand-designing the inputs the classifier will see: word counts, word lengths, whether a word is capitalized, whether the text contains an exclamation mark. The engineer guesses which signals matter, then the model learns weights on them. Feature engineering was hard and error-prone: the guess was often wrong, and a feature that helped on one task hurt on another. That gap is exactly what the next era filled.

1.2.3 Deep Learning, Transformers, and the Agentic Era

Deep learning arrived precisely because feature engineering was a bottleneck: instead of designing features by hand, the network learns features on its own. Give a deep network raw text, and the early layers learn the useful signals — word patterns, then phrase patterns, then sentence patterns. Then came the breakthrough of word-to-vector representation — word2vec-style embeddings that turned words into dense numeric vectors (the same word-to-vector mapping from the previous section, now at scale). After that, the transformer era: the attention mechanism and everything built on it. And today, the agentic AI era, where language models stop being answer-machines and start acting — planning multi-step tasks, calling tools, and checking their own work.

The honest footnote from industry: not everyone is actually running agentic AI. It is compute-intensive and expensive. Practitioners report that agentic systems only make sense when there is a return on investment — otherwise it is, in the instructor's own analogy, like using a rocket to go to the market. The right tool depends on the application, and the course will keep returning to that judgment call.

1.2.4 Why History Matters: Old Fundamentals Still Run Today

History is not trivia here. Research progress is mostly gap-filling: you study what exists, find the issues, and work on those. It is rarely the apple-falls-on-your-head moment. So the lineage matters because each era solved the previous era's bottleneck — no data led to rules, no features led to deep learning — and the same pattern is repeating in the agentic era.

Here is the whole ladder in one table, with the resource each era had and the wall it hit:

Read the last column top to bottom: each era exists because it removed the previous era's wall. When the agentic era eventually gets expensive and opaque enough, the same gap-filling logic says the next era will target cost and transparency.

The Fourier series analogy is the instructor's favorite proof that old ideas stay alive: invented long ago, and still used in speech recognition, in multimodal models, and in signal processing for images. Fundamentals from the James Allen era survive the same way. This session, the instructor promised, is the easiest one of the course — the trailer before the film.

1.3 Applications of NLP

Before any theory, the tour of what NLP actually does. These are the systems you already touch daily — the point is to notice them, and to see how many of them are the same machinery in different clothes.

1.3.1 Speech, Healthcare, and Everyday Automation

Speech recognition is now basic plumbing. The Whisper API can transcribe you in a noisy market, with background noise cancellation built in. Whisper is OpenAI's speech-to-text model: audio in, written text out, trained on a huge mix of languages and accents, and exposed as a simple paid API — the same pattern as the other commercial NLP APIs this session surveys. Text-to-speech is equally easy — it can mimic a lecturer's voice so well that voice deep fakes are now a serious, separate challenge. Notice the asymmetry: making the tools easy also made the abuse easy, which is why the safety applications later in this section matter.

Real-world: healthcare. India is a high-population country with queues everywhere — hospitals, insurance, appointments. FAQ agents, triage agents, and conversational AI can speed all of that up. A triage agent is the first contact a patient talks to: it sorts cases by urgency — booking routine appointments itself, and routing chest pain to a human doctor immediately. Sorting incoming cases by severity is what hospital staff call triage, and it is one of the highest-value places to put a conversational system, because it moves the queue instead of just answering it. Real-world: e-commerce too — Amazon-style question answering systems handle customer queries the same way.

1.3.2 Database Dialogue, Code Agents, and Search

One of the most practical industrial applications is the dialogue interface to databases. Around 80% of industry data sits in databases of various kinds. Instead of learning SQL, joins, foreign keys and primary keys, you write a question in natural language, the system queries the backend, and it can even convert the result back into natural language for you. There are many open-source and proprietary APIs for exactly this.

Why this matters: a database is only as useful as the number of people who can ask it questions. SQL is a wall — joins (combining rows from two tables), foreign keys (columns that point to rows in another table), and primary keys (columns that uniquely identify a row) are all concepts a business analyst may never learn. The NLP version moves the wall: question in, SQL out, rows back, then a natural-language summary of those rows.

Real-world: code agents are NLP applications too. "Write me code that does this — in Python, or Java, or whatever language" is natural language input that produces code. And search itself is NLP territory: in retrieval augmented generation (RAG), finding the relevant context out of a huge corpus is an NLP problem. RAG splits the job in two: a retriever finds passages that might answer the question, and a generator writes the answer using those passages as evidence. Both halves are NLP, and the course revisits RAG (and its knowledge-graph cousin GraphRAG) in a dedicated module.

1.3.3 Information Retrieval vs Information Extraction

These two sound alike and get confused constantly, so the contrast matters. Information retrieval is searching: given a topic, find the knowledge or the passages. Information extraction is different: pull out the structured pieces — named entities, relations, events. Given a paragraph, extraction answers: who is the person named here, what organization, what event?

Take one paragraph: "Tata Motors announced a new plant in Pune on Tuesday." IR, given the query "Tata plant", returns this paragraph (and others like it) ranked by relevance. IE fills a table: organization = Tata Motors, event = announcement, location = Pune, date = Tuesday. Same paragraph, two different products. When to pick which: IR when the user wants to read, IE when the user wants fields — and many pipelines run IR first, then IE over the retrieved passages.

It is harder than it looks. The name "Tata" versus "Ratan Tata Foundation" — one string could be a person, an organization, or a company depending on context. Real-world: Amazon's NLP tools do relationship extraction — "X is the capital of Y", or "this person teaches NLP at that university" — automatically. Amazon also sells PII (personally identifiable information) detection for guardrail use.

1.3.4 Writing, Translation, and Safety Applications

Real-world: QuillBot bundles plagiarism detection, a "humanizer" that rewrites AI text, an AI chat, an image generator, an automatic summarizer, and citation generation in IEEE or ACM formats, plus grammar checking and paraphrasing. Most of it is open for small use, paid at larger scale. Real-world: Grammarly has made spell check and grammar check part of daily life. The instructor's nostalgia lands here: in school, Wren and Martin grammar drills were torture, and now nobody memorizes past participles because tools do it. Those tools are NLP applications.

Real-world: machine translation. Traveling in regions that prefer local languages is easier when MT is strong, and languages are worth preserving anyway — so translation stays a key application. Real-world: cybersecurity. Cyberbullying detection is a critical, growing NLP application — cyber cells around the world use NLP to detect these issues, and fake news and plagiarism detection belong to the same family. They share one core trick: classification over text. The difference is only what the labels mean — "toxic / not toxic", "fake / real", "plagiarized / original".

1.3.5 Student Questions and Answers

1.4 NLP Tools and Libraries

Nobody builds NLP systems from zero. Every application in the previous section is assembled from pre-built pieces — and choosing which pieces is a core engineering skill in itself.

1.4.1 Commercial Tools

The giants sell NLP as APIs. Google Cloud NLP offers pre-trained models for entity analysis (similar to information extraction), sentiment analysis, and document classification. Amazon provides relationship extraction (part of entity and relation extraction) and PII detection for guardrails. Microsoft Azure is strong at enterprise deployments, especially in healthcare. IBM is used across industries for text understanding. All of them are paid.

The commercial model is simple to understand: you send text to a hosted endpoint, and you get structured results back — no GPUs, no model downloads, no setup. What you pay for is three things: the trained model, the infrastructure that runs it, and the service guarantees (uptime, compliance, support) that enterprises demand. What you give up: control, transparency about what happens inside, and money that scales with every API call.

1.4.2 Open-Source Libraries

The open-source side is where most learning happens:

• Hugging Face — the hub of language models, plus the Transformers and other libraries. Almost everyone in the field uses it.
• NLTK — the classic NLP toolkit. Open source, very easy, needs minimal resources, runs fast, and dominates education and research because it is free. You can explore a huge range of tasks in NLTK, and today it even has bindings for deep learning transformers. Bonus: it supports many Indian languages — stop words, POS tagging, and more exist for Hindi, Marathi, Tamil, Bengali, and others.
• spaCy — the industrial-strength alternative. Fast, production-oriented, and today optimized mainly for English (though it supports more languages).
• Stanford CoreNLP — the Java option for those working in Java stacks.
• TextBlob — a very easy library for many NLP tasks, popular in applications.

There are many more, but these are the ones industry actually reaches for.

Two rules of thumb cut through the catalog. First, paid API vs open-source library: pick the paid API when you want a hosted, maintained service with no infrastructure work and the budget exists; pick the open-source library when you need control, offline capability, zero per-call cost, or a language the APIs handle poorly. Second, heavy vs light: Hugging Face and spaCy carry real machine-learning models; NLTK and TextBlob are lighter, classical toolkits that run anywhere — the same lightness-vs-power trade the preprocessing section explores next.

1.5 Hands-On Text Preprocessing with NLTK and spaCy

Midway through the session, the class ran two small shared code notebooks (one NLTK, one spaCy) in Jupyter or Colab. They were deliberately simple — beginner level, runnable in seconds. The payoff is that every preprocessing step below is a real, runnable technique, not a diagram. Preprocessing is the front door of every NLP system: whatever you plan to do later — classify, retrieve, embed — the raw text must first be cut, cleaned, and normalized into pieces the machine can chew.

1.5.1 Tokenization: Splitting Text into Words and Sentences

Why care about subwords? This is the deep motivation. Words need to be split into smaller subword units so they can be stored and retrieved from memory efficiently. That is why every OpenAI-style bill is priced in tokens — tokens are the unit everything is measured in. GPT itself uses a more advanced tokenizer, byte pair encoding (BPE), rather than the simple space-based tokenizer shown here. BPE builds its cuts from data: it starts with characters and repeatedly merges the most frequent character pairs, so common pieces like "ing" or "tion" become single tokens. Later sessions will cover BPE, WordPiece, and SentencePiece properly, but the underlying concept is the same: split text into small reusable pieces.

Real-world: tokenization quality even explains model efficiency stories. When DeepSeek arrived, a big part of the "it is faster and cheaper" claim came down to a better tokenization technique — the same concept, executed better.

1.5.2 Punctuation Removal and Stop Words

After tokenization, the demo removed punctuation using the string library. Brackets, commas, and full stops may carry little meaning for many tasks — and every one of them still consumes a token. In RAG pipelines, where you retrieve context to reduce LLM hallucination, trimming useless punctuation shrinks token counts and costs.

Next, stop words. NLTK ships with a stop word list in its corpus — words like "down", "themselves", "be", "each", "you" — that may not carry much significance for the content. The demo removed words like "I", "am", "is", "are". You can also build your own domain-specific stop word list — different words matter in genomics than in finance — and NLTK lets you add custom lists.

Two practical bonuses make this technique valuable. First, it is cheap: for a startup or small organization, classical preprocessing on light hardware gives good, accurate results without burning GPU money. Second, it is highly explainable — there are no mysterious failures, and hallucination risk is basically zero in these steps. And yes, the same removal works for local languages — Hindi, Marathi, Tamil, Bengali, and also French and German.

1.5.3 Lemmatization and Stemming

Both techniques compress word forms back toward a shared base. Every word has a lemma — the root word — plus an affix or suffix. This is the same idea BPE uses: break the whole word into subparts. If "processing" becomes "process" + "ing", and "interested" becomes "interest" + "ed", then you store one embedding for "interest", one for "ing", one for "ed" — and you cover interested, interesting, interests, and more without storing each full word separately.

The two techniques sit on different ends of a trade-off:

That is the dictionary side. The next question pushes one level deeper, into the rule mechanics themselves.

The practical advice: there are many ready-made stemmers and lemmatizers — like sklearn, you don't write the algorithm from scratch — and you can try multiple stemmers and pick whichever gives the best results for your data.

1.5.4 Part-of-Speech Tagging and WordNet

NLTK also ships a POS tagger — the perceptron tagger, a neural-network-based library that labels each token with its part of speech: noun, verb, and so on. The demo output used tag codes like DT and JJ (determiners, adjectives, and the rest) — those will be explained in the dedicated POS tagging module later.

Two codes, decoded here so the demo output is readable: DT is a determiner (words like "the", "this" that point at a noun), and JJ is an adjective (words like "blue", "tall" that describe a noun). The dedicated POS module covers the full tag set; for now the point is that a code like JJ tells a downstream system "this word describes things, so it cannot be a person name".

The demo also touched WordNet — the lexical database of word meanings — but only in passing; it returns properly in the word sense disambiguation module. One preview of why it matters: WordNet organizes words into meaning relations — "cat" is a kind of "animal", "tire" is a part of "car" — and those relations are the raw material for resolving word senses. There are many ready-made POS taggers, open source and free, that need almost no compute.

1.5.5 NLTK vs spaCy: Student Questions and Answers

The spaCy notebook ran the same ideas — tokenization, stop words, lemmatization, POS tagging — and handled the same "Dr. X" example correctly. The visible differences: spaCy tokens also carry their character position in the text (which NLTK does not report), and spaCy's POS output spells things out in full ("proper singular noun") where NLTK gives codes. spaCy is faster; NLTK gives fewer extras.

On the lab side, one practical note: the virtual labs earn marks — using them for the programming assignments earns one or two marks — so they are worth the setup friction.

1.6 Why Language Is Hard: Ambiguity Everywhere

1.6.1 Sources of Difficulty

Language has rules, and rules have exceptions. English is famously irregular — sometimes "an" attaches before words that merely sound vowel-like: you say "an hour" (silent h, so it sounds vowel-initial) but "a university" (starts with the y-sound of "you"). Nobody taught you that rule explicitly; you absorbed it. And every language, from French to any Indian language, has its own thicket of rules and exceptions — each one a separate pile of traps for a machine.

Then there is ambiguity. The word "cool" meant something boring a generation ago and now means everything is fine. Gen Z uses words sarcastically, inverting meanings. Some things are culturally acceptable in one country and not another. New words keep entering the dictionary — words like "tandoori" and "yoga" have been absorbed into English. Add Gen Z jargon and emoticons, and the input is noisy. For humans this is all obvious and effortless. For machines, it is a very difficult training problem — because every one of those shifts breaks the assumption a machine is trained on: that words keep their meaning.

1.6.2 Worked Example: The Black Panther Booking Dialogue

A four-line dialogue with a conversational AI system shows how much a machine must do. Walk through it turn by turn.

None of that information needed to exist at training time. A modern agentic system reaches out at run time — external APIs for the theater, calculator access for the total — and composes an answer. But look at what four short lines demanded: named entity recognition, place disambiguation, coreference, temporal inference, constraint checking, and arithmetic. A human does it without noticing. That gap is the field.

1.6.3 How Humans Process This Dialogue

The instructor turned the question on the class: how did you know Black Panther was a movie and Mountain View was a place?

That is the uncomfortable part: you cannot point to the step where you decided "Black Panther is a movie". It was not a lookup in a dictionary, and it was not a grammar rule you applied consciously. Your knowledge base (everything you have seen and heard), your discourse history (what was said earlier in this conversation), the word "playing", and the familiar sentence order all voted together, instantly. A machine has to rebuild each of those votes from scratch — which is why the scrambled-sentence experiment works: break the word order, and the whole vote collapses, for humans too.

1.6.4 Word Order and Structural Ambiguity

Bag-of-words models make the point crisply. A bag-of-words model is exactly what it sounds like: the text is reduced to the set of words it contains, with every word's position thrown away — like shaking all the words out of a sentence into a bag. Compare two sentences:

A second classic: "Visiting relatives can be a nuisance."

The deeper point: the same word can have multiple parts of speech, and the meaning changes with the part of speech — exactly what happened with "visiting". This is structural ambiguity: the words are fixed, but the grammar tree can be built two ways, and each tree yields a different meaning.

1.6.5 Lexical Ambiguity and Changing Meanings

More everyday cases:

• "I go now." Translate this into an Indian language and you must commit to gender. In Marathi, "mī jāte" (मी जाते) marks a female speaker, "mī jāto" (मी जातो) a male one. English hides the information; the translation forces it out.
• "Can." A milk can or water can — a noun, a container. "Can I go?" — a modal verb. And a third meaning many people miss: to can is to pack something (as in canned food). One word, three parts of speech.
• "Bank." River bank or financial bank? Only context decides.

These ambiguities are not edge cases — they are the reason contextual word embeddings were a breakthrough. A fixed vector for "bank" cannot capture both senses; a context-dependent vector can. The closing quote of the session, from Khalil Gibran, says the same thing in one line: "Wisdom is not in words. Wisdom is meaning within the words."

1.7 The Six Levels of Language Understanding

Humans build language understanding in layers, starting from the very first words learned in childhood, and the field maps its work onto those layers. The stack runs: morphological, lexical, syntactic, semantic, pragmatic, discourse. Today's systems sit around the pragmatic level; discourse still has work to do. Each higher layer assumes the lower layers are already handled.

1.7.1 Morphological Knowledge

Morphological knowledge is about word structure. Every word can be built from a morpheme — the smallest meaning-carrying piece, which is the same thing as the lemma, the root word, from the lemmatization demo. A book like Word Power Made Easy (the classic GRE vocabulary book) teaches exactly this skill: if "-logy" means "the study of something", then biology is the study of life and bioinformatics is information about life. You can figure out unknown words by knowing the parts. BPE and subword tokenizers encode the parts of words for exactly this reason — they capture the affixes and suffixes, the reusable pieces.

The textbook version of this level sorts word-building into three processes: inflection (grammatical variations of one word: walk, walks, walking — same word, different grammatical clothes), derivation (making a new word class: wide → widely, weak → weaken), and compounding (joining two words: tea kettle, disk drive). All three matter to machines for one reason: language is productive. Any text you process will contain words nobody has listed in a dictionary, and the only way to handle them is to understand their parts.

1.7.2 Lexical Knowledge

Lexical knowledge is about which affixes attach to which words. It is beautiful, not "friendful". It is friendship, not "beautyship". Over years of exposure, your brain's neural network learned which suffix goes with which word. Lexical knowledge is the learned wiring of which combinations are legal.

Push the pattern one step and you feel the rule you carry: you accept "unhappy" but not "unsad"; "happiness" but not "angriness". There is no master list you memorized — your exposure to language wired in the frequencies. That is precisely the signal a machine can learn too: show it enough text, and it picks up that "friendship" happens and "friendful" never does. This level is the quiet bridge between the previous two: morphology says what the parts could build, lexical knowledge says which builds the language actually accepts.

1.7.3 Syntactic Knowledge

Syntactic knowledge is grammar — and every language has its own. English is SVO: subject, verb, object. "I eat mango." Indian languages are typically SOV: "Mī āmbā khātē" — the verb comes at the end. In Marathi the verb also agrees with the speaker's gender: "मी आंबा खातो" (mī āmbā khāto) for a male speaker and "मी आंबा खाते" (mī āmbā khāte) for a female speaker — the word order rule and the gender rule ride together. You would never say "I mango eat" in English, because you know the grammar of your language. And for complex sentences, wrong grammar makes meaning unrecoverable — you cannot even parse what was meant.

That last sentence is the reason syntactic knowledge sits below semantics in the stack: if the word order is scrambled past recognition, the meaning layer has nothing to build on. Grammar is not decoration; it is the load-bearing structure meaning stands on.

1.7.4 Semantic Knowledge

Semantic knowledge is where meaning lives. Contextual word embeddings, attention, and the whole semantic layer of modern NLP are trying to capture meaning. The demonstration sentence: "Green ideas have large noses." Grammatically perfect. Meaningless — ideas cannot be green, and ideas cannot have noses. You know the words "green" and "ideas" don't combine into meaning. That knowledge is semantic.

The classic version of this sentence is "Colorless green ideas sleep furiously" — grammatically flawless and semantically empty. Both versions make the same point: grammar can pass while meaning fails. A machine that only checks syntax would happily accept "green ideas have large noses" as a good sentence; a machine with real semantic knowledge must notice that nothing in the world is a green idea with a nose.

1.7.5 Pragmatic Knowledge

Pragmatic knowledge is meaning in the world, not in the dictionary.

This is the level where agentic AI systems struggle today. The connection to agent memory is direct: semantic memory holds what things are; episodic memory holds what happened; together they supply the world knowledge that says a pen does not cut bananas.

1.7.6 Discourse Knowledge

Discourse knowledge is cross-sentence connection — coreference stretched over distance. "She" refers back to Namrata across sentences, not just within one. The context window in an LLM is discourse knowledge in machine form: give a model 50,000 tokens of history and it can figure out meaning after 50,000 words. The childhood story example: "There was a king and a queen. ... They lived happily together." You know "they" means the king and queen, even paragraphs later. That is the context window you carry in your head.

Notice what makes discourse different from every lower level: nothing in a single sentence tells you who "they" is. The information lives in the history — in sentences already said. A model with a context window stores that history and consults it; a model without one faces every sentence alone, and "they" means nothing.

1.7.7 These Levels Inside LLMs and Agentic Systems

Modern agentic systems use all of these levels without advertising it. A conversational AI system that finds user intent is doing syntactic, semantic, and pragmatic analysis. Personalization uses personal history and goals — discourse and pragmatic knowledge in action. The reason contextual word embeddings were a breakthrough is exactly this: they capture the meaning of words through vector representation, and in the agentic era, capturing user intent is the same problem one level up. If the intent is wrong, the output is weird — and nobody wants weird outputs.

1.8 Anatomy of an NLP System

The applications, tools, and preprocessing steps all snap together here. This section opens the hood and shows what a complete NLP system looks like on the inside — and why the design that wins in production is neither the classical textbook diagram nor the pure transformer.

1.8.1 Understanding and Generation

Picture the classical pipeline diagram from the James Allen era: text enters on the left, passes through a row of boxes — input, syntactic analysis, semantic analysis, domain knowledge — and exits on the right as generated text. The exact boxes changed over the years, but the shape never did: understand in, generate out. That shape is the anatomy of every NLP system that has ever shipped.

1.8.2 Classical, Modern, and Hybrid Systems

The diagram from the James Allen era looks dated, but the blocks are the same; only the implementation differs.

The classical system: input, syntactic analysis, semantic analysis, domain knowledge, and generation — all explicit, hand-built stages. Each box is a separate program you can open, read, and debug. The cost is that every box had to be built by hand, which is why the classical systems were narrow and brittle.

The modern system: input, tokenization (text preprocessing), then a transformer that does syntax and semantics internally — with no transparency. It is not explainable; parameters and GPUs replace explicit steps. The transformer is a single box that learned to do what the classical row of boxes did, but nobody can point to where the syntax stage lives inside it.

The hybrid system is what production actually uses. Classical preprocessing (tokenization, stop word removal, lemmatization) replaces the expensive transformer for the basic steps. Then retrieval models — RAG tools, agentic flows, or vector databases for traditional RAG — provide context. Then the LLM generates. Then classical post-processing cleans up generation.

One sentence to pick: classical when the task is narrow and you need explanations; modern when the task is broad and you have GPUs; hybrid — the production default — when you want the cheap classical steps to do the cheap work and the LLM to do only the expensive, genuinely hard work.

1.8.3 Why Hybrid Systems Win in Production

The economics decide. Transformers need GPUs, and GPUs are expensive: an A800 or H800 — the "800-series" GPU the instructor quoted — runs about 5–6 lakh rupees, and an H100 costs more, around 12 lakhs. Four GPUs at 5 lakhs each is 20 lakhs — unaffordable for a startup doing basic text processing. Lighter classical preprocessing runs anywhere and gives good results for simple tasks.

The engineering argument is transparency. The preprocessing steps are highly explainable and their hallucination risk is basically zero.

Two final cost notes. One: agentic AI is resource-intensive; use it only when there is a return on investment, otherwise you are taking a rocket to the market. Two: prefer search engines over LLMs when you are just looking things up — LLMs have roughly 100 times the carbon footprint, meaning far more resources consumed.

1.9 Evaluating NLP Systems

The system is built. Now the uncomfortable question: is it any good? This final section turns evaluation into a checklist — the right metric, the right data, the right economics, and the ability to explain failures.

1.9.1 Why Evaluation Matters

1.9.2 Classification Metrics: Accuracy, Precision, Recall, F1

The session assumes you already know these from earlier ML/DL courses, so it does not derive them — but the standard forms are included here for completeness, because every later metric in this course leans on them. The starting point is the confusion matrix: four counts of what a classifier did. With true positives (TP, the correct hits), true negatives (TN, the correct rejections), false positives (FP, the false alarms), and false negatives (FN, the misses):

Accuracy and F1 dominate classification tasks like sentiment analysis and spam detection. Precision and recall matter most where errors are asymmetric — fraud detection and medical scenarios. The session's most memorable example: in cancer research, both false directions are dangerous. A false positive tells a healthy person they have cancer, and the mental damage is real. A false negative tells a sick person they are fine, and that can cost a life. In such scenarios you must control both, and the choice of metric is part of the design.

1.9.3 Task-Specific Metrics: BLEU, ROUGE, Perplexity

Different tasks need different yardsticks. BLEU is the standard for machine translation: it compares the system's translation against human reference translations by counting matching word sequences, with a penalty for very short outputs. ROUGE is used for text summarization: it measures how much of the reference summary's wording the system's summary recovers. Perplexity is the language modeling metric — it measures how good the LLM is, in one number: the inverse probability the model assigns to a test text, read as "how surprised is the model by the next word". Lower surprise means a better model. It will be covered properly in the language modeling module. (The Perplexity product many students know is a separate thing — the name is borrowed.) The later modules cover BLEU and ROUGE in detail; for now, know which metric belongs to which task.

1.9.4 Data Quality: Generalization and Fairness

The data decides the system. It must generalize — test performance must match training performance, with no overfitting, and it must hold on diverse, unseen datasets. Overfitting is the specific failure this guards against: the model memorizes the training examples instead of learning the pattern, so it scores well on the training set and collapses on anything new. It must be fair: no gender bias, no caste bias, nothing baked in. Bias is not a bug in the training code — it enters through the data, and once a biased pattern is in the training text, the model treats it as truth. Garbage in, garbage out — low-quality input data produces a system you cannot trust, and trust is fragile. One visible failure and users abandon the system, and in enterprise deployments a faulty model loses customers.

1.9.5 Efficiency: ROI, Latency, and Cost

Efficiency is an evaluation criterion too. Return on investment: if the infrastructure costs a crore of rupees, you expect at least two crores back. Latency matters more than people admit — measured as time to first token, the delay between your query and the first word of the answer appearing. If ChatGPT took ten minutes to produce its first result, you would stop using it. Users have lost their patience; results need to arrive within seconds, or at worst a minute or two. The system also has to scale — handling ten users is not the same as handling ten thousand, and an evaluation that ignores load is testing a demo, not a product.

1.9.6 Observability and Root-Cause Analysis

Finally, observability: can you reproduce a result by repeating the same steps? Do you know when failures happen and why? Which errors are most prominent, and what causes them? Root-cause analysis of failures is evaluation too — a system is not "evaluated" if you cannot say why it failed, only that it did. The scale problem makes this urgent: a large language model can have 145 billion parameters, a small language model 1 billion, some models a few million, and nano models around 100,000. With that many parameters, explaining why a system behaved a certain way is one of the biggest open challenges in AI — and it is why governance has become such a big issue.

Exam Guidance Summary

Everything the session said about how this course is graded and run, in one place.

• Weightage map. Two quizzes worth 5% total, with the best of the two counted. Experiential learning / lab assignment 1: 12%. One more assignment covering pre-mid-sem and post-mid-sem material: 13%. A closed-book makeup exam worth 30%, and an open-book end-semester (comprehensive) exam worth 40%.
• Deadlines are hard. There is no extension for any quiz or assignment. The dates have already been shared — mark them in advance.
• Virtual labs earn marks. Using the virtual labs for programming assignments earns one or two marks, so use them. If the labs are genuinely blocking (e.g., the VM idles out during long runs), a local machine or Google Colab is allowed as a fallback — but try the labs first, and report lab problems to the lab support team.
• Groups. Programming assignment groups may be formed across sections.
• Assumed knowledge. Accuracy, precision, recall, and F1 are assumed known from earlier ML/DL courses and will not be re-derived. BLEU, ROUGE, and perplexity will be covered in later modules (machine translation, summarization, language modeling respectively).
• Textbook. Roughly 80% of the course content comes from Jurafsky and Martin; the James Allen book supplies the fundamentals content of this session.
• Homework. At the students' request, short practice homework (not graded assignments) will be given after every session from the next session onward.

• AI tools are allowed, but contribute. You may use AI tools for assignments — but write your own work and understand it. Faculty can recognize the stylistic fingerprints of code and text generated by Copilot, ChatGPT, Claude, or Gemini, so outsourced work is detectable. The tools are a help, not a substitute.

Key Industry Applications

Consolidated list of every real-world connection made in this session:

• Real-world: Conversational AI and agents — booking systems (movie tickets), FAQ agents, customer support; systems combine named entity recognition, coreference resolution, and external API calls at run time.
• Real-world: Speech — Whisper API for speech recognition with noise cancellation; text-to-speech that can mimic voices; voice deep fakes as an emerging risk.
• Real-world: Healthcare — triage agents, FAQ agents, and conversational AI to cut hospital and insurance queues in high-population countries.
• Real-world: E-commerce — question answering systems for customer queries (Amazon).
• Real-world: Databases — natural language interfaces to databases, replacing SQL, joins, and keys; results converted back into natural language.
• Real-world: Code agents — natural language to code generation in any language.
• Real-world: Search and RAG — retrieval of relevant context; hybrid systems with vector databases; GraphRAG built on knowledge graphs (a knowledge graph is a populated ontology).
• Real-world: Information extraction — named entities, relations, events; ambiguity of names like Tata vs Ratan Tata Foundation; Amazon's relationship extraction and PII detection for guardrails.
• Real-world: Writing tools — QuillBot (plagiarism, humanizer, summarizer, citations in IEEE/ACM format), Grammarly (spell and grammar checking).
• Real-world: Machine translation — travel and local-language preservation; gender resolution required in Indian languages.
• Real-world: Safety — cyberbullying detection used by cyber cells worldwide, fake news detection, plagiarism detection.
• Real-world: Model economics — token-based pricing; DeepSeek's efficiency advantage from better tokenization; GPU costs (5–6 lakhs for an 800-series GPU, about 12 lakhs for an H100); hybrid classical-plus-LLM systems as the production norm; LLM carbon footprint about 100 times that of search engines.
• Real-world: Careers — named roles: NLP engineer, prompt engineer, data scientist, conversational AI engineer, computational linguist.

Read the list once more and notice the pattern: every row is one of the course modules wearing an industry costume. Conversational AI is tokenization, POS tagging, and named entity recognition chained together. Search and RAG is information retrieval plus embeddings. Databases is understanding and generation with a query in between. The session's whole argument lands here — the fundamentals are the applications.