DNN

Deep Neural Network Components and Perceptron

📅 Published: 2026-07-13

🎓 Level: postgraduate

👥 Audience: Postgraduate students in Deep Neural Networks

2.1 Core Components of a Deep Learning Problem

Hook: You have probably used a voice assistant. You say "Hey Siri" and your phone wakes up. How does a machine learn to do that? It was not programmed with an "if audio-waveform-looks-like-Hey-Siri then wake" rule. Somebody had to teach a machine from scratch — using nothing but lots of audio examples and a clever recipe. That recipe has exactly four ingredients.

2.1.1 Definition and Explanation

Intuition: Think about teaching a small child to tell animals apart. You show the kid many pictures — a dog, a cat, a bird, a fish. With each picture you say what it is. The pictures are the data. The names you attach to each picture are the labels. Over time the kid builds an internal mental model that can recognize new animals it has never seen before. Deep learning works the same way, with machines as the learner. The child's mental checklist ("does it have a fin? whiskers? a trunk?") is a rough version of what the machine builds internally.

Every deep learning problem has four core ingredients. They are: data, a model, an objective function, and an optimization algorithm. The objective function is also called a loss function. Understanding how these four pieces work together during training is the foundation of the entire course.

Here is the flow: you feed data into the model. The model makes a prediction. The loss function measures how wrong the prediction is. The optimization algorithm updates the model to reduce that error. This cycle repeats thousands of times. That is training.

2.1.2 Data, Features, and Labels

2.1.2.1 What makes up data

Data has two parts: the actual input and the label (or target). The input could be an image, a sound recording, or a table row. The label is the classification or value you want the machine to learn.

Take an image. What makes one animal different from another? Distinguishing characteristics. The presence or absence of a whisker, a long tail, a trunk, a fin. These are features — structured pieces of information that describe each object.

In machine learning and deep learning, the set of all features is denoted by a vector $X$ , broken into $X_{1}, X_{2}, X_{3}, \dots$ . So $X_{1}$ could be the presence of a fin. $X_{2}$ could be the presence of a tail. $X_{3}$ could be whiskers, and $X_{4}$ could be a trunk. Each object gets a binary vector: an elephant gets $[0, 1, 0, 1]$ (no fin, has tail, no whiskers, has trunk). A fish gets $[1, 1, 0, 0]$ (has fin, has tail, no whiskers, no trunk).

This vector representation matters because machines only understand numbers. The labels are also encoded numerically: dog = 1, cat = 2, fish = 3, elephant = 4. The set of all distinct labels is denoted by $Y$ , often called the target or response variable.

2.1.2.2 Features, instances, and explanatory variables

Each row in a dataset is one unique instance (one house, one image). The properties of each instance — number of bedrooms, location, square feet — are called explanatory variables. They are also called features or inputs. The column you want to predict (say, house price) is the target variable or response variable. When the target is a category, it is called a label. When it is a continuous number, it is just called the target.

Worked example — encoding animal features as vectors:

You have four animals and four binary features: fin, tail, whiskers, trunk. Build the feature matrix.

Animal	Fin $X_{1}$	Tail $X_{2}$	Whiskers $X_{3}$	Trunk $X_{4}$	Label $Y$
Dog	0	1	1	0	1
Cat	0	1	1	0	2
Fish	1	1	0	0	3
Elephant	0	1	0	1	4

Each row is one instance. The four feature columns form $X$ . The label column is $Y$ . Notice: the dog and cat share the same feature vector $[0, 1, 1, 0]$ . A machine using only these four features cannot tell them apart. It needs more discriminative features like ear shape or fur pattern. This shows why hand-coding features has limits. Deep learning discovers richer features on its own from raw pixels.

Sense-check: With only these four crude features, the machine would confuse dog and cat every time. The model is only as good as the features it gets.

Now compare the two feature-engineering philosophies:

Approach	Who designs features	Risk	Fix
Machine learning	Human data scientist	Bias — missing critical features or adding noise	Domain expertise
Deep learning	The algorithm itself	Needs much more data to work	Add more data

Q: During learning, there would be millions of features for all the pictures — how do we handle that?

A: This is exactly what distinguishes machine learning from deep learning. In machine learning, you hand-code features as a data scientist. In deep learning, the machine discovers what matters without you defining it upfront.

2.1.3 Models and Weights

2.1.3.1 What is a model

After the kid sees thousands of labeled pictures, the kid builds an internal logic. When you deploy the kid — show a new picture — it says "cat." That internal logic is a model.

At its simplest level, a model is a set of weights. These weights are numbers. They tell the machine how much importance to give each feature.

For identifying a fish, the kid learns:

$W_{1}$ — importance of fin (high)
$W_{2}$ — importance of tail (moderate)
$W_{3}$ — importance of whiskers (low, maybe zero)
$W_{4}$ — importance of trunk (low, maybe zero)

Some textbooks use the symbol $θ$ (theta) instead of $W$ . Both mean the same thing: a weight parameter that the machine learns. The set of all weights is the model — also called the parameters of the network or, in machine learning, the hypothesis.

Q: Does a higher weight mean the feature is more significant?

A: Yes. The weights are what the machine actually learns. A high weight for "fin" means the model leans heavily on that feature. A weight near zero means the model ignores that feature.

2.1.3.2 The Maggi analogy for understanding weights

You are sitting in the living room. Someone is cooking in the kitchen. Your nose catches a smell — it reminds you of Maggi noodles. That smell is one input. You also hear sizzling — a second input. Neither alone is enough. Your brain combines them, giving more weight to smell (say 5 units) and less to sound (say 1 unit). Together you infer "probably Maggi."

But the same masala smell could be for another dish. So you walk to the kitchen. Your eyes see something yellow in a vessel. Now vision gets its own weight. Smell plus color together, with appropriate weights, give you enough confidence to say "yes, Maggi is cooking."

The key insight: these weights generalize. The same set of weights works for telling Maggi from dosa from pulav. It works as long as the machine was trained on all three. The machine extracts a pattern that generalizes across all the classes it has seen.

2.1.4 Objective Functions and Loss

When the machine predicts "cat" but the answer is "tiger," there is an error. The error is the difference between the predicted output $Y^$ and the actual target $Y$ .

The entire goal of training is to keep reducing this error. So we define an objective function — specifically a loss function — that quantifies how wrong the current model is. The machine adjusts its learning so the loss keeps decreasing.

We do not say "maximize accuracy." We say "minimize the loss." The objective is to reach the smallest possible loss.

Worked example — computing a simple loss:

Imagine a model that predicts house prices. For three houses, you have:

House	Actual price ( $Y$ )	Predicted price ( $Y^$ )	Absolute error $∣ Y - Y^∣$
A	50 lakh	45 lakh	5 lakh
B	80 lakh	90 lakh	10 lakh
C	60 lakh	60 lakh	0 lakh

A simple loss function — mean absolute error (MAE) — averages these errors:

$Loss = \frac{5 + 10 + 0}{3} = \frac{15}{3} = 5 lakh$

The loss is 5 lakh rupees per house on average. The optimizer's job: tune the model weights so this number gets smaller. A perfect model would have loss = 0.

Sense-check: House B had the largest error (10 lakh). The optimizer would nudge weights to pull that prediction down from 90 toward 80 on the next pass.

Scope — what a loss function does not do: A loss function only measures error on the data it sees. It does not automatically tell you whether new, unseen data will also have low error. That gap — between training error and generalization error — is the central problem of machine learning. You will see it again and again in this course.

Q: Is the objective function the same as the loss function? Is limiting mispredictions its only job?

A: Yes, the objective function is a loss function. Its job is to make the loss as small as possible. But the loss is not limited to just counting mispredictions. For classification, you might use cross-entropy loss. For regression, mean squared error or mean absolute error. Each is designed for a different problem type.

Q: What if I give new data at runtime that does not match any training data?

A: Once deployed, a model is fixed — it does not learn from new data. Over time, data distribution changes (new customer tastes, new infrastructure). The model becomes stale. This triggers retraining: gather new data, augment the dataset, retrain, redeploy. This field is called MLOps. You can also build monitors that flag when model performance drops below a threshold. If users keep giving thumbs-down for two months, it is time to retrain.

Q: Is there a numeric threshold for the loss — like "this much loss = good model"?

A: It depends on your criteria. For perfect accuracy, let training run until loss hits zero. But that may never happen. Often you stop training when the loss stops changing. If the loss stays at 0.5 for 100 iterations without dropping, you might decide to stop. Both approaches — hard thresholding and plateau detection — are valid.

2.1.5 The Training Process: Learning Algorithms and Optimization

The learning algorithm is a sequence of steps through which the machine learns. A popular algorithm for deep learning is gradient descent, covered in later sessions.

2.1.5.1 Optimization — not just more data

If the model is not accurate, adding more data is one fix. But sometimes the machine learns too fast or too slow, or skips over important patterns. The ways you control how the model learns are collectively called optimization.

The learning algorithm drives optimization. It controls how the model adjusts its weights, how fast it takes each step, and when it stops. You — the developer — choose the algorithm and tune its settings. The combination of setups is what makes the algorithm run. What algorithm suits which scenario, what parameters make it work better — this is the art of learning.

Q: Does the algorithm only exist for fine-tuning?

A: No. The algorithm is a core ingredient throughout training, redeployment, and retraining. Learning happens through a series of steps — that is the algorithm. It is present any time a model learns.

Q: Does the developer choose the learning algorithm?

A: Yes. Within each algorithm there are settings you can tune — parameters to change, others to reduce, others to keep constant.

2.1.6 Visual Intuition — The Four-Ingredient Loop

Picture a circular flow diagram:

Data enters from the left — rows of feature vectors with labels.
The Model (a box in the center) takes the data and outputs predictions $Y^$ .
The Loss Function (a gauge below the model) compares $Y^$ to $Y$ and shows a number — the error.
The Optimization Algorithm (an arrow looping back from loss to model) adjusts the weights to reduce the error.

This loop runs over and over. Each pass: predict → measure error → adjust weights → predict again with slightly better weights. After thousands of loops, the loss drops low enough. The model is trained.

2.1.7 Symbol Registry — Core Components

Symbol	Meaning	LaTeX	Type	Domain
$X$	Feature vector	$X$	vector	$R^{d}$
$X_{i}$	i-th feature	$X_{i}$	scalar	varies
$Y$	Target / label set	$Y$	varies	categorical or $R$
$Y^$	Predicted output	$Y^$	scalar	varies
$W_{i}$ or $θ_{i}$	Weight for feature i	$w_{i}$ or $θ_{i}$	scalar	$R$
Error	Difference $Y - Y^$	$Y - Y^$	scalar	$R$
Loss	Objective function value	—	scalar	$R_{\geq 0}$

2.1.8 Wake Word Example — Training a Model End to End

Consider training a machine to recognize a wake word like "Hey Alexa." The sound signal varies with different speakers and different amplitudes. The raw waveform is high-dimensional and unstructured. Hand-coding features is practically impossible.

Training phase:

Collect millions of audio recordings.
Label each as "wake word" (1) or "not wake word" (0).
The machine trains on this labeled data.
It cross-verifies predictions against labels. If wrong, the optimizer adjusts weights.
More data → better generalization in complex deep-learning problems.

Inference (deployment) phase:

A new sound signal arrives at the microphone.
The trained model processes it.
The model's generalized pattern says "yes, this is a wake word."
The device wakes up.

The same architecture can be tuned to recognize "Hey Siri" or "Hey Cortana." This is the power of generalization. The model extracts a pattern for "a wake word" that works for any specific wake word it sees during deployment.

Pitfalls:

Confusing features with labels. Features are inputs ( $X$ ). Labels are the answers you teach ( $Y$ ). They are different things. A common beginner mistake: including the label as a feature.
Thinking "more data always fixes everything." More data helps, but a bad model design or poor optimization settings will still fail. The optimizer matters as much as the data.
Believing training loss tells the full story. A model with near-zero training loss might fail on new data (overfitting). Training loss is a progress report, not a guarantee. Always check performance on data the model has not seen.
Expecting loss to reach exactly zero. In real problems, zero loss almost never happens. A loss that plateaus at a low, stable value is a healthy signal — the model has converged.

Recap: Every deep learning problem breaks into four pieces — data, model, loss function, and optimizer. The model learns by cycling through them thousands of times, each pass driven by the learning algorithm.

Bridge: You now know the ingredients list. The next section groups these recipes into three families. They are based on what kind of data you feed in — supervised, unsupervised, and reinforcement learning.

Real-world connection: Wake word detection ships in hundreds of millions of devices worldwide (Amazon Echo, Apple HomePod, Google Nest). The core engineering challenge is not just accuracy. It is running a deep neural network continuously on a low-power chip inside a speaker. The goal: catch every "Hey Siri" but keep false triggers minimal. This four-ingredient pipeline also trains models for medical image diagnosis, fraud detection, and self-driving perception systems.

2.2 Supervised, Unsupervised, and Reinforcement Learning

Hook: Every machine learning model needs data to learn. But what if you have no labels? What if you have no data at all — just a problem and a goal? These three scenarios define three totally different ways machines learn. The one you pick changes everything: what you feed in, how you measure success, and what you can expect to get out.

Intuition: Think of three students preparing for an exam. The first student gets a textbook with all answers filled in. She studies the question-answer pairs and learns to solve similar questions. That is supervised learning. The second student gets only the questions, no answers — she groups them by topic on her own and discovers patterns. That is unsupervised learning. The third student gets nothing. She tries solving past papers, gets scored, and adjusts her strategy after every try. That is reinforcement learning.

Symbol Registry — Learning Types

Symbol	Meaning	LaTeX	Type	Domain
$X$	Feature vector (input data)	$X$	vector	$R^{d}$
$Y$	Target / label set	$Y$	varies	categorical or $R$
$(X, Y)$	Labeled training pair	$(X, Y)$	pair	varies

2.2.1 Supervised Learning

When you provide labeled data — both the features and the correct target — you are supervising the learning process. The machine compares its predictions against your labels and adjusts. This is supervised learning. The "supervision" is the label — it tells the machine the right answer so it can correct itself.

House price example: Features are number of bedrooms, location, square feet. Target is the price (a continuous number). The machine learns from historical records where the price is already known.

Animal classification example: Features are fin, tail, whiskers, trunk. Labels are dog (1), cat (2), fish (3), elephant (4). Here the target is categorical — a discrete class.

Deep learning can be applied to supervised learning tasks. These are the tasks you will focus on for most of this course.

2.2.2 Unsupervised Learning

In unsupervised learning, you provide data without labels. The machine must find inherent, naturally occurring patterns on its own. You might feed in property data — bedrooms, location, square feet, and price. Then you ask the machine to segment the properties into three groups. But you do not tell the machine what the groups should be. The machine discovers correlations invisible to human eyes and clusters instances accordingly.

This clustering activity — grouping data into distinguishing groups without predefined labels — is unsupervised learning. Deep learning can be applied to unsupervised tasks. More on this is covered in an advanced deep learning elective.

2.2.3 Reinforcement Learning

What if you have no data at all? This is reinforcement learning — learning by directly interacting with an environment through trial and error. The only signal is feedback from the environment — rewards or penalties.

Example: A kid learning to ride a bicycle. The kid pedals, falls, gets a bruise (penalty), adjusts, tries again. After many trials, the kid learns to balance. There was no pre-existing dataset — just interaction and feedback.

Deep learning applied to reinforcement learning is the core of a second-semester course on deep reinforcement learning.

Worked example — matching the learning type to the scenario:

Scenario	Data available?	Labels?	Learning type
Predict house price from historical sales	Yes (bedrooms, location, sq ft)	Yes (actual sale price)	Supervised (regression)
Group customers into three segments based on buying patterns	Yes (purchase history)	No	Unsupervised (clustering)
A robot learns to walk in simulation	No — only reward signals	Only feedback	Reinforcement learning
Classify emails as spam or not spam	Yes (email text)	Yes (spam/not spam)	Supervised (classification)
Discover topics in a collection of news articles	Yes (article text)	No	Unsupervised (topic modeling)

Sense-check: The dividing factor is "do you have the right answer for every example?" Yes = supervised. No, but you have inputs = unsupervised. No data, just reward signals = reinforcement.

Here are the three learning types compared side by side:

Dimension	Supervised	Unsupervised	Reinforcement
Input	Features + labels $(X, Y)$	Features only $(X)$	Environment state
Goal	Learn a mapping from $X$ to $Y$	Discover hidden structure in $X$	Learn a policy that maximizes reward
Feedback	Immediate (label tells right/wrong)	None (no answer key)	Delayed (reward after sequence of actions)
Example tasks	Classification, regression	Clustering, dimensionality reduction	Game playing, robotics, autonomous driving
Data cost	High (labeling is expensive)	Low (no labels needed)	Zero upfront cost but many interactions needed

When to pick which: Use supervised learning when you have a clean labeled dataset. Use unsupervised when you have data but no labels and want to explore structure. Use reinforcement learning when you cannot get a labeled dataset but can simulate or interact with an environment that gives feedback.

Q: Can we use reinforcement learning for all problems? Finding labeled data is hard.

A: You can use reinforcement learning for any problem if two conditions hold. First, the environment must give a feedback signal for each action. Even delayed feedback works. Second, the task must be goal-oriented. But reinforcement learning can be costly. If you already have relevant, up-to-date data — like Bangalore house prices still valid today — supervised learning is much faster. If good labeled data exists, use it.

2.2.4 Classification vs. Regression Tasks

Within supervised learning, tasks fall into two categories based on the type of target:

Classification: The target is a discrete category. If you classify objects into predefined groups (cat, dog, fish), you have a classification task. Two classes = binary classification. More than two = multi-class classification.

Regression: The target is a continuous numerical value. Predicting house price answers "how much" — this is a regression task.

Multiple tasks can coexist. An object-detection system must classify the object — building, human, bag, bicycle, pushcart. It must also do regression — finding XY coordinates and bounding box width/height. Classification answers "which?" Regression answers "how much?"

Pitfalls:

Confusing unsupervised and reinforcement learning. They are not the same. Unsupervised has data but no labels. Reinforcement has no data — only an environment that gives rewards. If someone says "no labels," ask: "do you have any input data at all, or just a task and a reward signal?"
Treating ordinal categories as regression. If your target is a rating from 1 to 5 stars, that is ordinal classification. The gap from 1 to 2 may not equal the gap from 4 to 5.
Assuming classification needs balanced classes. It does not. You can have 99% class A and 1% class B. The model will struggle, but it is still a valid classification problem. You handle imbalance with techniques like class weighting.

Recap: Three families of learning — supervised (labeled data), unsupervised (data without labels), and reinforcement (no data, only environmental feedback). Within supervised learning, tasks split into classification (discrete target) and regression (continuous target).

Bridge: Now that you know how data shapes the learning type, the next section asks: what properties must the data itself have? And when does deep learning beat traditional ML?

Real-world connection: Self-driving cars use all three paradigms. Supervised learning classifies traffic signs and pedestrians from labeled camera images. Unsupervised learning clusters driving scenarios to discover rare edge cases. Reinforcement learning trains the car's policy — steering, braking, accelerating — in simulation. The reward is "stay on the road and avoid collisions." One vehicle, one problem domain, all three learning types working together.

2.3 Properties of Data and When to Use Deep Learning

Hook: Most machine learning textbooks start with a quiet assumption — that every data point is independent of every other. But what if today's stock price depends directly on yesterday's? What if your data has a million dimensions? These two situations — sequential dependence and high dimensionality — break traditional ML. They are exactly where deep learning shines.

2.3.1 Independent and Identically Distributed (IID) Data

Most traditional machine learning algorithms assume the data is independent and identically distributed (IID). Each instance (each row) is independent of every other. All instances come from the same underlying distribution.

Independent: The house price of one property does not directly cause the price of another. Each row stands alone.

Identically distributed: All data points are drawn from the same statistical pattern. Data from a single locality like Bangalore follows the same distribution.

Analogy: Think of rolling a fair die 100 times. Roll 37 does not change the outcome of roll 38. Each roll is independent. And every roll comes from the same distribution (each face has probability 1/6). The rolls are IID. Most traditional ML models — linear regression, decision trees, SVM — were built with this mental model. Each dataset is a big table of independent rows drawn from one population.

2.3.2 Sequential Data and Autocorrelation

Many real-world problems break the IID assumption. Consider stock prices: the share price on Day 1 influences Day 2, which influences Day 3. The data points are autocorrelated — each entry depends on previous entries. This is sequential data or time-series data.

Traditional machine learning struggles with autocorrelation because it treats each row as independent. Deep neural networks, specifically recurrent neural networks (RNNs), can handle sequential patterns. RNNs are covered in the post-midterm part of this course.

More examples of sequential data applications:

Automatic speech recognition: A sentence is a sequence of words. The first word influences the second. You cannot recognize one word in isolation.
Natural language processing (NLP): Machine translation, text summarization, text-to-speech, speech-to-text — all process sequences of natural language text.

Worked example — distinguishing IID from sequential data:

Suppose you have two datasets:

Dataset A (IID): 10,000 Bangalore house sale records. Each row: bedrooms, location, square feet, sale price. Row #42 (a house in Koramangala) does not depend on row #43 (a house in Whitefield). They are independent.

Dataset B (Sequential): 1,000 daily closing prices of Reliance stock over 4 years. The price on Day 5 depends partly on Day 4, which depends on Day 3. If you shuffle the rows randomly, the model loses the temporal pattern entirely.

Property	Dataset A (Housing)	Dataset B (Stocks)
Rows independent?	Yes	No
Row order matters?	No	Yes
Model type	Traditional ML or DNN	RNN / LSTM / Transformer
IID?	Yes	No

Sense-check: Shuffle the housing data — the model still learns. Shuffle the stock data — the model fails. That test tells you if your data is IID.

2.3.3 High-Dimensional Data and Automatic Feature Extraction

In machine learning, a data scientist manually hand-codes features. This introduces bias — you might miss critical features or include unnecessary ones. In deep learning, the machine automatically extracts features. You give it raw data (pixels, audio samples) and the network discovers the distinguishing characteristics on its own.

This is critical when:

The input is very high-dimensional (thousands or millions of features).
The data is noisy and hand-coded features are unreliable.
You want features to be automatically curated.
The number of features grows exponentially with task complexity.

Object detection example: Hand-coding features to distinguish objects in a traffic camera feed is impractical. You would need a huge binary feature vector. Let the machine extract features automatically.

Deep learning should also be preferred when interpretability is not important. Traditional ML models (like decision trees or linear regression) have better explainability. If you must explain every decision, trade accuracy for interpretability and use traditional ML.

Here is a decision guide:

Scenario	Recommendation
Very high-dimensional input (thousands of features)	Use DNN
Data is noisy, feature definitions are unclear	Use DNN
Want automatic feature extraction	Use DNN
Complex sequential / time-series patterns	Use DNN (RNN)
Target function is unknown and data is complex	Use DNN
Interpretability of model decisions is critical	Use traditional ML
Pattern is simple, features are well-structured	Traditional ML suffices

Scope — the IID assumption:

Violation #1 — Autocorrelation. If rows depend on each other (time series), IID is broken. Traditional methods will underestimate variance and overstate confidence. Use sequential models (RNN, LSTM) instead.
Violation #2 — Distribution shift. Even if rows are independent, training data and test data may come from different distributions. Data collected in 2019 may not match 2024. The model fails silently. This is called covariate shift. Monitor and retrain.
Violation #3 — Non-identically distributed. Your training data may mix Bangalore houses and New York apartments. The two populations have different distributions. The model learns a confused average. Split by region or use a model that handles multiple distributions.

Pitfalls:

Assuming all real-world data is IID. It often is not. Financial data, speech, text, video — these are all sequential or structured. Check the IID assumption before picking your model.
Using deep learning for a simple problem with clean features. If you have 10 features and 500 rows, a decision tree or linear regression will work better.
Confusing high-dimensional with "lots of rows." DNNs help when features (columns) are many, not when rows (samples) are many. If you have 10 features and a billion rows, traditional ML scaled with distributed computing may be the right call.

Recap: Two data properties decide the tool: (1) IID vs. sequential — sequential data needs RNNs; (2) low vs. high-dimensional — high-dimensional, noisy data with unclear features calls for deep learning. When interpretability matters more than accuracy, stick with traditional ML.

Bridge: With the what and when of deep learning in hand, we now zoom into the smallest building block — the perceptron. It is a mathematical model of a single brain cell, and it is where everything starts.

Real-world connection: In finance (the BFSI domain), stock price prediction systems must handle sequential autocorrelated data. A traditional linear regression would treat each day's closing price as independent — it would miss the trend entirely. RNNs, LSTMs, and transformers are the standard tools. The same tools power automatic speech recognition in call centers — transcribing millions of customer calls. They also power machine translation in tools like Google Translate and DeepL.

2.4 The Perceptron — Biological Inspiration

Hook: Your brain has about 86 billion neurons. Each one is a tiny decision maker. It receives a burst of signals, sums them up, and decides "fire" or "don't fire." In 1957, Frank Rosenblatt built a mathematical model of exactly one such neuron. He called it the perceptron. It could answer only yes or no. It output only 1 or 0. But that single yes/no cell is the ancestor of every deep neural network running today.

2.4.1 Biological Neuron Structure

Intuition: Nature inspires much of AI. Birds inspired airplanes. Ant behavior inspired optimization algorithms. The human brain inspired the artificial neural network. The parallel is direct. A biological neuron gets signals through dendrites. It processes them in the cell body (soma). It sends the result down an axon. An artificial neuron gets features, computes a weighted sum, and outputs 1 or 0.

The basic computational unit of the brain is the neuron. A neuron receives signals from multiple sensory inputs through dendrites. The cell body (soma) processes these signals — combining and associating the information. The processed signal travels to other neurons or body parts through axons, surrounded by a myelin sheath.

When the same pattern is observed repeatedly, the axon strengthens its response — the neuron "remembers" the pattern. This strengthening of connections is the biological basis of learning. Computationally, this translates into the weights — they store the learned information.

The brain has about $1 0^{11}$ neurons with roughly $1 0^{15}$ interconnections. Neuron switching time is on the order of milliseconds. Yet the brain recognizes complex scenes in under a second. It does this through massive parallel processing — different neurons process different inputs at the same time, then combine their outputs.

2.4.2 The Artificial Neuron (Perceptron) Model

The perceptron is the simplest artificial neural network, invented in 1957 by Frank Rosenblatt. It is a mathematical abstraction of a single biological neuron. It was designed for binary classification — answering yes or no, outputting 1 or 0.

A perceptron receives multiple inputs $X_{1}, X_{2}, \dots, X_{n}$ (features). Each input $X_{i}$ is multiplied by a corresponding weight $W_{i}$ (or $θ_{i}$ ). The perceptron sums all these products:

$W_{1} X_{1} + W_{2} X_{2} + \dots + W_{n} X_{n} = i = 1 \sum n W_{i} X_{i}$

Then it adds a bias term $b$ (also written as $W_{0}$ with $X_{0} = 1$ ):

$z = b + i = 1 \sum n W_{i} X_{i}$

This consolidated value $z$ is passed through a threshold or activation function. The simplest activation checks whether $z \geq 0$ . If yes, the neuron fires (outputs 1). If not, it outputs 0.

$f (z) = {1, 0, if z \geq 0 if z < 0$

Notation note: Textbooks often write the perceptron as $y^= f (w^{⊤} x + b)$ or $y^= f (θ^{⊤} x)$ . Here we use $W_{i}$ for each weight and $b$ for the bias, matching the lecture. The symbols $W_{i}$ and $θ_{i}$ mean the same thing: a learned weight parameter.

Worked example — computing a perceptron with numbers:

You have a perceptron with two inputs and learned weights:

$W_{1} = 2.0$ , $W_{2} = - 1.0$ , $b = - 0.5$

An input arrives: $X_{1} = 1$ , $X_{2} = 1$ .

Step by step: $z = b + W_{1} X_{1} + W_{2} X_{2}$ $z = - 0.5 + (2.0) (1) + (- 1.0) (1)$ $z = - 0.5 + 2.0 - 1.0$ $z = 0.5$

Since $z = 0.5 \geq 0$ , the neuron fires: output = 1.

Now try another input: $X_{1} = 0$ , $X_{2} = 1$ .

$z = - 0.5 + (2.0) (0) + (- 1.0) (1)$ $z = - 0.5 + 0 - 1.0$ $z = - 1.5$

Since $z = - 1.5 < 0$ , output = 0.

$X_{1}$	$X_{2}$	$z$	Output
1	1	0.5	1
0	1	-1.5	0
1	0	1.5	1
0	0	-0.5	0

Sense-check: This perceptron outputs 1 whenever $X_{1} = 1$ (with $W_{1} = 2.0$ providing strong positive pull) regardless of $X_{2}$ . The negative $W_{2}$ and negative bias create a high bar — only when $X_{1}$ pulls the sum above zero does it fire.

2.4.3 Symbol Registry — Perceptron

Symbol	Meaning	LaTeX	Type	Domain
$X_{i}$	i-th input feature	$X_{i}$	scalar	varies
$W_{i}$ (or $θ_{i}$ )	Weight for input i	$W_{i}$ or $θ_{i}$	scalar	$R$
$b$ (or $W_{0}$ )	Bias term	$b$ or $W_{0}$	scalar	$R$
$X_{0}$	Bias input (always 1)	$X_{0} = 1$	scalar	${1}$
$z$	Weighted sum + bias	$z = b + \sum W_{i} X_{i}$	scalar	$R$
$f (z)$	Activation / threshold output	$f (z) \in {0, 1}$	scalar	${0, 1}$

2.4.4 Bias — Meaning and Significance

2.4.4.1 Application intuition

Consider a model that predicts house price:

$Price = W_{1} \cdot (square feet) + W_{2} \cdot (location score) + W_{3} \cdot (transport score) + b$

If someone asks: "Without knowing square feet, location, or transport, what is the baseline house price?" The bias $b$ answers that question. It is the base price that exists even when all features are zero or unknown.

2.4.4.2 Graphical intuition — the AND gate example

Consider the AND gate truth table: output is 1 only when both inputs $X_{1}$ and $X_{2}$ are 1. All other combinations give 0.

Plot these data points on a 2D graph with $X_{1}$ on the x-axis and $X_{2}$ on the y-axis:

$(0, 0)$ at origin → output 0
$(1, 0)$ to the right → output 0
$(0, 1)$ above → output 0
$(1, 1)$ at top right → output 1

The perceptron learns a decision boundary — a line that separates the 1-point from the 0-points. This decision line follows:

$b + W_{1} X_{1} + W_{2} X_{2} = 0$

Without a bias term ( $b = 0$ ), the decision boundary equation becomes $W_{1} X_{1} + W_{2} X_{2} = 0$ , which forces the line to pass through the origin $(0, 0)$ . That severely restricts what patterns the neuron can learn. The bias shifts the decision boundary away from the origin, giving the neuron flexibility.

Worked example — AND gate with a perceptron:

We need weights and bias such that:

$(1, 1)$ → output 1
$(0, 0), (1, 0), (0, 1)$ → output 0

One valid set: $W_{1} = 1$ , $W_{2} = 1$ , $b = - 1.5$ .

Check each point:

$(1, 1) : z = - 1.5 + (1) (1) + (1) (1) = - 1.5 + 2 = 0.5 \geq 0$ → output 1 ✓

$(1, 0) : z = - 1.5 + (1) (1) + (1) (0) = - 0.5 < 0$ → output 0 ✓

$(0, 1) : z = - 1.5 + (1) (0) + (1) (1) = - 0.5 < 0$ → output 0 ✓

$(0, 0) : z = - 1.5 + 0 + 0 = - 1.5 < 0$ → output 0 ✓

Decision boundary: $- 1.5 + X_{1} + X_{2} = 0$ → $X_{1} + X_{2} = 1.5$ . This is a line shifted away from the origin. If $b = 0$ , the line would be $X_{1} + X_{2} = 0$ , which passes through $(0, 0)$ and cannot separate the AND points.

Sense-check: The bias $b = - 1.5$ pushes the line so that $(1, 1)$ falls on the positive side and all other points on the negative side. Without it, AND separation is impossible.

Several students asked about the bias term:

Q: Is bias the same as the intercept $C$ in $Y = m X + C$ ?

A: Yes. The bias $b$ is the intercept term. In English, "bias" can mean partiality — that is not the meaning here. Technically, bias is the intercept that shifts the decision boundary.

Q: Does the machine learn the bias, or is it fixed?

A: The machine learns it. During training, the bias is adjusted along with all other weights. It is part of the model's parameters.

Q: Can we have multiple biases?

A: Each neuron gets one bias. In a single perceptron, there is one bias. In a multi-layer network, each neuron has its own bias.

2.4.5 Thresholding and Activation Functions

After computing the weighted sum $z = b + \sum W_{i} X_{i}$ , the perceptron checks it against a threshold. The simplest threshold is 0:

If $z \geq 0$ , output 1 (neuron fires).
If $z < 0$ , output 0 (neuron does not fire).

The threshold can be any value — 0.5, 0.7, 0.8 — depending on the confidence level you want. The activation function is a transformation unit that takes the input $z \in (- \infty, + \infty)$ and maps it to a desired output range:

Some activation functions map to $(0, 1)$ .
Others map to $(- 1, + 1)$ .

In the simplest binary case, the activation answers: "Is the signal strong enough to fire?"

Q: Is the threshold automatically decided during learning?

A: In the early lectures, the threshold is hardcoded — you set it to 0. Later, with multi-layer networks, activation functions (like sigmoid or ReLU) learn the right transformation automatically. The network adjusts its activation behavior during training.

2.4.6 Multi-Layer Perceptrons — When One Perceptron Is Not Enough

A single perceptron can only draw a linear decision boundary — a straight line in 2D, a flat plane in 3D. This works for AND and OR. But it fails for XOR, where the output is 1 when exactly one input is 1.

In XOR, the points $(0, 1)$ and $(1, 0)$ are labeled 1, while $(0, 0)$ and $(1, 1)$ are labeled 0. On a 2D plot, the 1-points are diagonally opposite each other. No single straight line can separate them — the problem is nonlinearly separable.

Analogy: Imagine you need to fence off two sheep standing in opposite corners of a square field. With one straight fence, you cannot enclose just the two sheep without also enclosing one of the other corners. You need two straight fences placed at different angles, then combine their enclosed areas. That is what an MLP does — first layer draws two lines, second layer combines them.

The solution: use more than one perceptron. Build two perceptrons, each learning its own decision line. Then connect their outputs to a third perceptron that combines them — creating a nonlinear decision boundary. This is the idea behind multi-layer perceptrons (MLP). Stacking perceptrons in layers captures complex patterns that a single perceptron cannot.

Worked visualization — XOR with two decision lines:

Perceptron A learns: "fire when $X_{1} + X_{2} > 0.5$ " (separates $(1, 1)$ from others)
Output: 1 for $(1, 1)$ , 0 for $(0, 0), (1, 0), (0, 1)$
Perceptron B learns: "fire when $X_{1} + X_{2} < 1.5$ " (separates $(0, 0)$ from others)
Output: 1 for $(0, 0), (1, 0), (0, 1)$ , 0 for $(1, 1)$

Neither A nor B alone can solve XOR. But combine them. A second-layer perceptron learns the combination rule: "output 1 when A=0 and B=1." This yields the XOR truth table.

Sense-check: Two lines working together can carve out a non-convex region. One line alone cannot. That is depth.

2.4.7 Student Questions and Answers

The professor addressed several student questions during this session. These are grouped by confusion point:

2.4.7.1 On the decision boundary and origin

Q: If both $X_{1}$ and $X_{2}$ are 0 at the origin, what happens?

A: The data point $(0, 0)$ falls on one side of the decision boundary. For AND gate logic, all points on that side are labeled 0. The decision boundary is what the model extracts from the data. At $(0, 0)$ , it gets label 0 — correct for AND.

2.4.7.2 On notation consistency

Q: In the perceptron equation we write $W_{0} X_{0}$ with bias, but earlier you wrote $b$ . Are they the same?

A: Yes. $W_{0}$ is the bias weight and $X_{0}$ is always 1. So $W_{0} X_{0} = b$ . Some textbooks use $b$ , others use $W_{0}$ . Both mean the same thing.

2.4.7.3 On feature representation and pixel inputs

Q: Could each feature — fin, tail, whiskers, trunk — be $X_{1}, X_{2}, X_{3}, X_{4}$ ?

A: Yes. You can design binary features and feed them into the network. You can also go deeper. Instead of hand-coding features, give the network the raw picture. Let it discover features on its own. A picture is a grid of pixels, and each pixel value can be one input unit.

2.4.7.4 On internal connections in multi-layer networks

Q: In a multi-layer network, do neurons within the same layer talk to each other?

A: No. Within one layer, neurons do not have direct connections. They operate in parallel, processing features independently. Their outputs are combined in the next layer. This is highly parallel and distributed processing — like your ears and eyes working simultaneously before the brain combines their signals.

2.4.7.5 On handling hybrid/conflicting features

Q: If I have a hybrid animal like a liger (lion + tiger), do I need a separate perceptron?

A: Every perceptron helps detect one type of distinguishing pattern. The network as a whole learns to capture subtle distinctions. It adjusts all weights and biases across multiple neurons automatically. You do not manually wire connections. The training process adjusts the biases. The machine learns to separate patterns even when parts of patterns overlap across classes.

2.4.8 Properties of Artificial Neural Networks

Many neurons work in harmony. Each is a threshold switching unit. It decides whether received information is strong enough to pass forward.

Weighted interconnections. Every connection between neurons carries a weight. These weights (plus biases) are the parameters the machine learns. They are the model.

Highly parallel and distributed. Within a layer, every perceptron processes features simultaneously. They do not depend on other neurons in the same layer. The outputs are combined in the next layer — like ears and eyes working at the same time, with the brain combining their signals.

Automatic weight tuning. The network learns by adjusting weights through backpropagation and gradient descent algorithms (covered in later sessions).

Scope — assumptions of the perceptron model:

Binary features in the basic model. The simplest perceptron assumes inputs are numerical (binary 0/1 or real numbers). It cannot directly handle categorical text labels ("cat," "dog") — those must be encoded as numbers first.
Linearly separable data (for a single perceptron). A single perceptron converges only if the data is linearly separable. If you feed it XOR-like data, it will never settle on a solution — it will oscillate forever.
Deterministic output. The basic perceptron gives the same output for the same input every time. It does not model uncertainty or probability. Later activation functions (sigmoid, softmax) add probabilistic outputs.

Pitfalls:

Forgetting the bias. A perceptron without a bias ( $b = 0$ ) forces the decision boundary through the origin. This drastically limits what patterns it can learn. Always include a bias unless you specifically want an origin-crossing boundary.
Expecting one perceptron to solve everything. A single perceptron can only draw straight lines. Real-world data is rarely linearly separable. You need multiple layers for complex patterns.
Confusing weights with features. The weight $W_{i}$ is NOT the feature $X_{i}$ . The feature is the data. The weight is what the machine learns to assign as importance to that feature. Think of the feature as the "what" and the weight as the "how much it matters."
Applying the perceptron rule directly to continuous regression. The basic perceptron outputs 0 or 1 — it is a binary classifier. To predict a continuous value (like house price), you drop the threshold step and output $z$ directly — this becomes a linear regressor.

Recap: The perceptron is a weighted sum of inputs, plus a bias, passed through a threshold. It is a binary classifier inspired by brain neurons. One perceptron = one straight decision line. Stack many = multi-layer perceptron, which can learn non-linear patterns like XOR.

Bridge: Now we zoom out to see how many neurons organized into layers create a deep neural network — and what design choices the developer must make.

Real-world connection: The perceptron was the first hardware neural network. The Mark I Perceptron machine (1957) used physical potentiometers as weights. Electric motors adjusted them during training. Today, the same perceptron equation runs on modern deep learning chips like GPUs and TPUs. The math has not changed. What changed is the scale. From one perceptron in 1957 to billions of artificial neurons in 2024.

2.5 Deep Neural Networks — Architecture and Hyperparameters

Hook: If one perceptron is a single decision-maker, what happens when you wire a thousand of them together? You get a deep neural network. It transforms raw pixels into "this is a cat" through layer after layer. But here is the surprising part. Nobody can give you a formula for how many layers to use or how wide each should be. You have to figure that out yourself.

2.5.1 Layers and Feed-Forward Structure

A deep neural network organizes neurons into layers:

Input layer: Receives the raw features $X_{1}, X_{2}, \dots, X_{n}$ . Each neuron here holds one feature value. This layer does no computation — it just passes values forward.
Hidden layers: Intermediate layers between input and output. Neurons in these layers detect different patterns and pass their results forward. "Hidden" means you do not directly observe their outputs — they are internal to the network.
Output layer: Produces the final prediction — a class label, a number, or a probability distribution.

Information flows in one direction only — from input to hidden to output. This is a feed-forward neural network (FFNN). There are no loops, no backward connections, and no skipping layers. Every neuron in layer $L$ connects to every neuron in layer $L + 1$ (fully connected).

Analogy: Think of an assembly line in a factory. Raw material (features) enters at station 1. Each station (hidden layer) transforms it a bit. Station 1 detects edges. Station 2 assembles edges into shapes. Station 3 recognizes whole objects. The finished product pops out at the end (output layer). Material only moves forward — never backward.

In the human brain, connections are far more complex. There are recursive connections and self-loops — a neuron feeding its output back to itself. While such designs can be modeled, they are complex. For the first 13 sessions of this course, the focus is on feed-forward networks only.

Q: What is a self-loop?

A: A self-loop is when a perceptron takes its own output and feeds it back as one of its inputs. This happens in biological neural networks but is not part of the feed-forward model covered in early sessions.

2.5.2 What Makes a Network "Deep"

The term deep neural network (DNN) has different definitions across textbooks:

Some define it as any network with three or more hidden layers (not counting input and output layers).
Others say anything with more than one hidden layer qualifies.
A commonly cited threshold: more than three hidden layers makes a network deep.

Regardless of the exact number, "deep" means many layers. Each layer learns increasingly abstract patterns:

Early layers: detect simple features (edges, corners, color blobs in an image).
Middle layers: combine simple features into parts (ears, eyes, wheels).
Deep layers: combine parts into whole objects (face, car, building).

2.5.3 The Need for Depth — Complex Patterns and Digit Recognition

Consider recognizing a handwritten digit "7." The digit is not one atomic pattern. It consists of sub-patterns: a horizontal top stroke and a diagonal stroke. The digit "2" shares some of these sub-patterns. A "7" and a "2" have common features like horizontal strokes. They also have distinguishing ones — like the curve at the bottom of 2.

A single perceptron cannot capture these overlapping, hierarchical patterns. Multiple perceptrons — each detecting different sub-patterns — collaborate. The machine learns which features are common across digits and which are unique. It knows which perceptrons to emphasize and which biases to tune.

Key limitation: You cannot easily say "this perceptron detects exactly the top bar of 7." Interpretability is a hard problem in deep networks. Unlike a decision tree where you can trace every split, deep networks are black boxes. This is the accuracy-vs-explainability trade-off.

2.5.4 Hyperparameters

Designing a network involves choices that are not learned from data but must be set by the designer. These are hyperparameters:

Number of layers: How deep the network is.
Number of neurons per layer: How wide each layer is.
Learning rate: How big a step the optimizer takes. Too big = overshoot. Too small = takes forever.
Activation function: Which transformation to use after each weighted sum (sigmoid, ReLU, tanh).

There is no fixed rule for these choices. No rule says "for image processing, use 4 layers with 5 neurons." No rule says "for speech recognition, use 1 layer with 15 million neurons." These choices are empirically determined through experimentation.

Analogy: Think of building a house. The number of rooms and dimensions are architectural choices — hyperparameters. The furniture, paint color, and decor are learned during training — those are parameters (weights). You decide the layout. The occupants (data) determine the furnishings.

Worked example — hyperparameter choice and its effect:

You build a classifier for handwritten digits (MNIST, 10 classes). Input: 28×28 = 784 pixel values. Output: 10 classes. Try three architectures:

Architecture	Hidden layers	Neurons/layer	Total params	Train accuracy	Test accuracy
A (shallow)	1	32	~25K	92%	91%
B (medium)	2	128	~118K	98%	97%
C (deep)	5	256	~1M	99.9%	97.5%

Architecture B works best. It is deep enough to learn digits but not so large that it overfits. Architecture C gets near-perfect training accuracy but barely beats B on test data. Architecture A is too simple — it underfits.

Sense-check: More layers and neurons give more capacity and more overfitting risk. The best architecture balances capacity against overfitting. There is no formula — you must experiment.

2.5.5 When to Choose Deep Neural Networks Over Traditional ML

Scenario	Recommendation
Very high-dimensional input (thousands of features)	Use DNN
Data is noisy, feature definitions are unclear	Use DNN
Want automatic feature extraction	Use DNN
Complex sequential / time-series patterns	Use DNN (RNN)
Target function is unknown (unsupervised) and data is complex	Use DNN
Interpretability of model decisions is critical	Use traditional ML
Pattern is simple, features are well-structured	Traditional ML suffices

2.5.6 Symbol Registry — DNN Architecture

Symbol	Meaning	LaTeX	Type	Domain
$X_{i}$	i-th input feature to the network	$X_{i}$	scalar	varies
$L$	A layer in the network	$L$	integer	$N$

Pitfalls:

Over-engineering the architecture. Beginners often add too many layers "just to be safe." This makes training slow and increases overfitting. Start simple. Add complexity only when the simple model underfits.
Confusing parameters with hyperparameters. Parameters (weights, biases) are learned from data. Hyperparameters (layers, learning rate, activation) are set by you. If the model learns it from data, it is a parameter. If you pick it before training, it is a hyperparameter.
Using the same hyperparameters for every problem. The best architecture depends on data type. Images need CNNs. Text needs RNNs/transformers. Tabular data needs MLPs. Match the architecture to the data.
Expecting hyperparameters to fix bad data. No layer tuning fixes mislabeled data, missing values, or a dataset that is too small. Clean and size your data first.

Recap: A deep neural network is a stack of layers in a feed-forward flow. Depth lets it learn hierarchies of patterns. Hyperparameters — layers, neurons, learning rate, activation — are designer choices set before training. There is no formula: you experiment, observe, and tune.

Bridge: That completes the conceptual foundation of the course. The next section lists what the professor flagged as most examinable. The section after connects every concept to real-world applications.

Real-world connection: The feed-forward architecture is the backbone of recommendation systems. These models decide what Netflix shows you next. They suggest what Amazon recommends. They choose what YouTube plays after your video. These systems take your watch history as features. They pass it through multiple hidden layers. They output a ranked list of recommendations. Companies like Meta and Google employ teams whose full-time job is tuning hyperparameters. A single well-chosen learning rate or layer count can translate to millions of dollars in more engagement.

Exam Guidance Summary

This lecture introduces the foundational vocabulary and mental model for the entire course. Expect conceptual questions on:

Exam note — what to prioritize:

The four core components: data, model, objective/loss function, optimization algorithm.
The difference between supervised, unsupervised, and reinforcement learning — with examples for each.
Classification vs. regression — given a scenario, identify which type of task applies.
The perceptron model: formula $z = b + \sum W_{i} X_{i}$ , role of weights, and role of bias (know both intuitions: application-level baseline and graphical decision-boundary shift).
Why bias matters: without it, the decision boundary is forced through the origin.
The AND gate as a worked perceptron example — be able to compute it with numbers.
Why a single perceptron fails on XOR (nonlinear separability).
Four properties of artificial neural networks.
Definition of deep neural network (3+ hidden layers is the common threshold).
What hyperparameters are (layers, neurons per layer, learning rate, activation).
When to prefer deep learning over traditional ML and vice versa.

Exam note — what is NOT in this lecture:

Mathematical and numerical examples of perceptron computation with logic gates will come in the next lecture. This lecture builds conceptual understanding only.
Labs and programming sessions begin from the fourth contact session onward.
Gradient descent, backpropagation, ReLU, sigmoid, softmax — all covered in later sessions. Know they exist, but do not expect numerical problems on them yet.

Key Industry Applications

Every concept in this lecture maps to a product or system you have already used:

Wake word detection: Hey Alexa, Hello Siri, Hey Google — audio signal processing with deep learning. The same architecture generalizes across different wake words. Ships in hundreds of millions of smart speakers and phones worldwide. The core challenge: running a deep neural network continuously on a low-power embedded chip.

Object detection in traffic cameras: Multi-class classification (building, human, bag, bicycle, pushcart) plus regression (bounding box coordinates). A single application combines both classification and regression. Deployed in smart city infrastructure, autonomous driving perception, and retail analytics.

Stock price prediction (BFSI domain): Sequential time-series data with autocorrelation — requires recurrent neural networks (RNNs). Used by quantitative trading desks, hedge funds, and robo-advisors to model market trends and execute trades.

Natural language processing (NLP): Machine translation, text summarization, text-to-speech, speech-to-text. All involve sequence learning. The meaning of a word depends on the words before it.

Reinforcement learning: Used when no pre-existing labeled dataset exists. The agent learns by interacting with an environment through trial and error. Powers game-playing AI (AlphaGo, OpenAI Five), robotic control, autonomous navigation, and recommendation systems that learn from user click feedback.

MLOps: Continuous deployment and monitoring pipeline. Triggers retraining when model performance drops due to changing data. The deployment lifecycle from Section 2.1.4 is a core MLOps workflow. Model becomes stale. Gather new data. Retrain. Redeploy. This cycle runs at companies like Uber, Netflix, and Spotify.

DNN Lecture 02 notes · Deep Neural Network Components and Perceptron

Deep Neural Networks· postgraduate· 2026-07-13

Sections Breakdown

1Core Components of a Deep Learning Problem

The four ingredients — data, model, loss function, optimization — and how they form the training loop.

2Supervised, Unsupervised, and Reinforcement Learning

The three learning paradigms and the split between classification and regression.

3Properties of Data and When to Use Deep Learning

IID vs sequential data, high dimensionality, and automatic feature extraction.

4The Perceptron — Biological Inspiration

The artificial neuron: weighted sum, bias, threshold, and why one perceptron fails on XOR.

5Deep Neural Networks — Architecture and Hyperparameters

Layers, feed-forward flow, depth, and the hyperparameters a designer must choose.

6Exam Guidance Summary

What the professor flagged as most examinable in this lecture.

7Key Industry Applications

Real products that use each concept: wake words, object detection, trading, NLP, RL, MLOps.

Postgraduate students in Deep Neural Networks

Exam Revision Notes

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review — but if something doesn't make sense, go back to the full explanation in the main content.

The Four Core Components of Deep Learning

Must-know: Every deep learning problem has four ingredients: data, a model, an objective (loss) function, and an optimization algorithm. Training is the loop: predict, measure error, update weights, repeat.

$Loss = \frac{1}{n} i = 1 \sum n ∣ Y_{i} - Y^_{i} ∣$ where $Y_{i}$ is the true value, $Y^_{i}$ the model's prediction, and $n$ the number of samples.

Top pitfall: Confusing features $X$ with labels $Y$ . Never feed the label in as a feature.

Self-check: Name the four ingredients and say what the optimization algorithm does on each training pass.

Connects to: Supervised vs unsupervised learning, Loss functions, The perceptron model

Supervised Learning

Must-know: You train on labeled data — both features $X$ and the correct target $Y$ . The label lets the model correct itself after each prediction.

$Learn a mapping f : X \mapsto Y$ where $X$ is the feature vector and $Y$ the label.

Top pitfall: Thinking unsupervised and supervised are the same. Supervised needs labels; unsupervised has none.

Self-check: Given historical house sales with prices, is predicting price supervised or unsupervised? Why?

Connects to: Unsupervised learning, Classification vs regression, Loss functions

Unsupervised Learning

Must-know: You feed data without labels. The model finds hidden structure — clusters and patterns — entirely on its own.

$Find hidden structure in X alone (no Y)$ where $X$ is the feature set.

Top pitfall: Assuming every task needs labels. Clustering and dimensionality reduction need none.

Self-check: You have purchase histories but no customer segments. Which learning type fits?

Connects to: Supervised learning, Reinforcement learning

Reinforcement Learning

Must-know: No dataset — the agent learns by interacting with an environment and getting reward or penalty feedback for its actions.

$Learn a policy π that maximizes expected reward R$ where $π$ is the policy and $R$ the cumulative reward.

Top pitfall: Assuming RL solves everything. If good labeled data already exists, supervised learning is faster and cheaper.

Self-check: A robot learns to walk in simulation from reward signals alone. Which learning type is this?

Connects to: Unsupervised learning, When to use deep learning

Classification vs Regression

Must-know: Classification predicts a discrete class (cat/dog). Regression predicts a continuous number (price). Both are supervised tasks.

$Classification: y^\in {1, \dots, K}; Regression: y^\in R$ where $y^$ is the prediction and $K$ the number of classes.

Top pitfall: Treating an ordinal rating (1-5 stars) as regression. It is ordinal classification.

Self-check: Is 'spam or not spam' classification or regression?

Connects to: Supervised learning, Loss functions

Independent and Identically Distributed (IID) Data

Must-know: Traditional ML assumes each row is independent and drawn from the same distribution. Violations break those models.

$X_{1}, X_{2}, \dots, X_{n} \sim iid P (X)$ where each $X_{i}$ is one instance and $P (X)$ the shared distribution.

Top pitfall: Assuming all real-world data is IID. Financial, speech, and text data are usually sequential.

Self-check: Are 10,000 house records from one city IID? What about daily stock prices?

Connects to: Sequential data and autocorrelation, When to use deep learning

Sequential Data and Autocorrelation

Must-know: When rows depend on previous rows (time series), the data is autocorrelated and not IID. Use RNNs, LSTMs, or Transformers.

$X_{t} depends on X_{t - 1}, X_{t - 2}, \dots$ where $X_{t}$ is the value at time $t$ .

Top pitfall: Shuffling time-series data. It destroys the temporal pattern the model needs.

Self-check: If you randomly shuffle stock prices, can a traditional model still learn the trend?

Connects to: IID data, When to use deep learning

High-Dimensional Data and Automatic Feature Extraction

Must-know: Deep learning automatically extracts features from raw, high-dimensional, noisy data — no hand-coding by a human required.

$raw input X DNN learned features ϕ (X)$ where $ϕ$ is the function the network learns to map raw input to features.

Top pitfall: Using deep learning when features are few and clean — a decision tree or linear model wins then.

Self-check: You have a million pixels per image. Hand-code features, or use a DNN?

Connects to: When to use deep learning, Deep neural network architecture

The Perceptron Model

Must-know: A perceptron computes a weighted sum of inputs plus a bias, then applies a threshold to output 0 or 1.

$z = b + i = 1 \sum n W_{i} X_{i}, y^= {10 z \geq 0 z < 0$ where $X_{i}$ is input $i$ , $W_{i}$ its weight, $b$ the bias, $z$ the pre-activation sum, and $y^$ the output.

Top pitfall: Confusing the weight $W_{i}$ (learned importance) with the feature $X_{i}$ (the data).

Self-check: With $W_{1} = 2, W_{2} = - 1, b = - 0.5$ and input $(1, 1)$ , does the perceptron fire?

Connects to: Bias in the perceptron, Multi-layer perceptrons, Thresholding and activation

Bias in the Perceptron

Must-know: The bias $b$ is the intercept that shifts the decision boundary away from the origin. Without it, only origin-crossing lines are possible.

$b + W_{1} X_{1} + W_{2} X_{2} = 0$ is the decision boundary; $X_{1}, X_{2}$ are inputs, $W_{1}, W_{2}$ weights, and $b$ the bias that shifts it.

Top pitfall: Forgetting the bias. A perceptron with $b = 0$ cannot solve the AND gate.

Self-check: For AND with $W_{1} = W_{2} = 1$ , what bias lets $(1, 1)$ fire but not $(0, 0)$ ?

Connects to: The perceptron model, Multi-layer perceptrons

Multi-Layer Perceptrons and XOR

Must-know: One perceptron draws only a straight line. XOR is not linearly separable, so you need multiple layers to combine lines into a non-linear boundary.

$XOR is non-linearly separable \Rightarrow needs \geq 2 layers$ where XOR outputs 1 when exactly one of its two inputs is 1.

Top pitfall: Expecting a single perceptron to solve XOR. It will oscillate forever and never settle.

Self-check: Why can't one straight line separate the XOR points $(0, 1), (1, 0)$ from $(0, 0), (1, 1)$ ?

Connects to: The perceptron model, Deep neural network architecture

Deep Neural Network Architecture

Must-know: A DNN stacks input, hidden, and output layers in a feed-forward flow. Depth lets it learn hierarchies of features.

$input \to hidden_{1} \to \dots \to hidden_{L} \to output$ where $L$ is the number of hidden layers.

Top pitfall: Over-engineering — adding layers 'just in case' slows training and causes overfitting.

Self-check: What are the three layer types in a feed-forward network, in order?

Connects to: Hyperparameters, Multi-layer perceptrons, High-dimensional data

Hyperparameters

Must-know: Hyperparameters (layers, neurons per layer, learning rate, activation) are set by you before training. Parameters (weights, biases) are learned from data.

$hyperparameters: {L, n_{ℓ}, η, ϕ}$ where $L$ is layer count, $n_{ℓ}$ neurons per layer, $η$ the learning rate, and $ϕ$ the activation function.

Top pitfall: Confusing hyperparameters with parameters. If the model learns it from data, it is a parameter.

Self-check: Is the learning rate a hyperparameter or a parameter?

Connects to: Deep neural network architecture, When to use deep learning

Previous LectureLecture 01 · Session 1: Introduction and Overview of Deep Neural Networks

Next LectureLecture 03 · Perceptron Learning and Introduction to Regression

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

2.1 Core Components of a Deep Learning Problem

2.1.1 Definition and Explanation

2.1.2 Data, Features, and Labels

2.1.2.1 What makes up data

2.1.2.2 Features, instances, and explanatory variables

2.1.3 Models and Weights

2.1.3.1 What is a model

2.1.3.2 The Maggi analogy for understanding weights

2.1.4 Objective Functions and Loss

2.1.5 The Training Process: Learning Algorithms and Optimization

2.1.5.1 Optimization — not just more data

2.1.6 Visual Intuition — The Four-Ingredient Loop

2.1.7 Symbol Registry — Core Components

2.1.8 Wake Word Example — Training a Model End to End

2.2 Supervised, Unsupervised, and Reinforcement Learning

2.2.1 Supervised Learning

2.2.2 Unsupervised Learning

2.2.3 Reinforcement Learning

2.2.4 Classification vs. Regression Tasks

2.3 Properties of Data and When to Use Deep Learning

2.3.1 Independent and Identically Distributed (IID) Data

2.3.2 Sequential Data and Autocorrelation

2.3.3 High-Dimensional Data and Automatic Feature Extraction

2.4 The Perceptron — Biological Inspiration

2.4.1 Biological Neuron Structure

2.4.2 The Artificial Neuron (Perceptron) Model

2.4.3 Symbol Registry — Perceptron

2.4.4 Bias — Meaning and Significance

2.4.4.1 Application intuition

2.4.4.2 Graphical intuition — the AND gate example

2.4.5 Thresholding and Activation Functions

2.4.6 Multi-Layer Perceptrons — When One Perceptron Is Not Enough

2.4.7 Student Questions and Answers

2.4.7.1 On the decision boundary and origin

2.4.7.2 On notation consistency

2.4.7.3 On feature representation and pixel inputs

2.4.7.4 On internal connections in multi-layer networks

2.4.7.5 On handling hybrid/conflicting features

2.4.8 Properties of Artificial Neural Networks

2.5 Deep Neural Networks — Architecture and Hyperparameters

2.5.1 Layers and Feed-Forward Structure

2.5.2 What Makes a Network "Deep"

2.5.3 The Need for Depth — Complex Patterns and Digit Recognition

2.5.4 Hyperparameters

2.5.5 When to Choose Deep Neural Networks Over Traditional ML

2.5.6 Symbol Registry — DNN Architecture

Exam Guidance Summary

Key Industry Applications

DNN Lecture 02 notes · Deep Neural Network Components and Perceptron

Sections Breakdown

Exam Revision Notes

The Four Core Components of Deep Learning

Supervised Learning

Unsupervised Learning

Reinforcement Learning

Classification vs Regression

Independent and Identically Distributed (IID) Data

Sequential Data and Autocorrelation

High-Dimensional Data and Automatic Feature Extraction

The Perceptron Model

Bias in the Perceptron

Multi-Layer Perceptrons and XOR

Deep Neural Network Architecture

Hyperparameters

Was this lecture useful?

BitsNotes AI Assistant

Configure AI Chat

Sign in to use AI Chat