Introduction to Statistical Methods

Time Series Analysis — Foundations and Basic Forecasting Models

Published: 2026-07-07

Level: postgraduate

Audience: Postgraduate students in Introduction to Statistical Methods

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

Covariance and Correlation — covered in Lectures 12 and 13
Linear Regression — covered in Lecture 13
Time Series Components (Introduction) — covered in Lecture 13

Time Series Analysis — Foundations and Basic Forecasting Models

14.1 Multicollinearity

Why does adding more variables sometimes make a model worse, not better? You would think more information always helps. But when two predictors carry the exact same information, they confuse the model instead of helping it. That is the multicollinearity trap.

14.1.1 Definition and Intuition

Think of a meeting where two colleagues always say the exact same thing, in the exact same words, at the exact same time. You only need one of them in the room. The second adds zero new information — they just make the conversation noisier and harder to follow. That is what multicollinearity does to a regression model.

Now map this to math. Your model is $Y$ against $X_{1}$ and $X_{2}$ . If $X_{1}$ and $X_{2}$ move together so tightly that you cannot tell which one is driving the change in $Y$ , you have multicollinearity. The model cannot separate their individual effects. The coefficients become unstable — small changes in the data produce wildly different coefficient estimates.

The analogy breaks where real data is messier than two people saying the exact same words. In practice, $X_{1}$ and $X_{2}$ are rarely perfectly correlated — they are highly correlated, say 0.95. That is still enough to cause trouble.

Multicollinearity is the condition where two or more predictor variables in a regression model are so highly correlated that the model cannot reliably estimate their individual effects on the response variable $Y$ . The fix is simple: drop one of the collinear variables. Keep either $X_{1}$ or $X_{2}$ , not both.

14.1.2 Detecting Multicollinearity with VIF

You detect multicollinearity using the Variance Inflation Factor (VIF). The VIF measures how much the variance of a coefficient is inflated because of correlation with other predictors.

The formula:

VIF_{j} = \frac{1}{1 - R _{j}^{2}}

where $R_{j}^{2}$ is the $R^{2}$ you get when you regress predictor $X_{j}$ against all the other predictors in the model. If $X_{j}$ is nearly a linear combination of the other predictors, $R_{j}^{2}$ is close to 1, and the VIF blows up.

Decision rule:

VIF > 5: warrants investigation. Multicollinearity may be inflating standard errors.
VIF < 5: no serious concern. Proceed with the model.

Some texts use VIF > 10 as the hard cutoff. The principle is the same: a high VIF means the coefficient's variance is inflated, so hypothesis tests and confidence intervals become unreliable.

The remedy, regardless of how you detect it, is always the same: remove one of the correlated variables. You can also combine them (e.g., take an average) or use regularization (which we cover next).

Worked Example: Computing VIF

Suppose your model has three predictors: $X_{1}$ , $X_{2}$ , and $X_{3}$ . You regress $X_{1}$ against $X_{2}$ and $X_{3}$ and get $R_{1}^{2} = 0.85$ . Then:

VIF_{1} = \frac{1}{1 - 0.85} = \frac{1}{0.15} \approx 6.67

A VIF of 6.67 exceeds the threshold of 5. Multicollinearity is likely inflating the standard error of the coefficient for $X_{1}$ . Investigate — check the correlation matrix between $X_{1}$ , $X_{2}$ , and $X_{3}$ , and consider dropping one.

Sense-check: A high $R_{j}^{2}$ in the auxiliary regression means the other predictors can almost perfectly predict $X_{j}$ — which is exactly what makes the coefficient unstable.

Scope: Where VIF applies and where it breaks.

Applies to: linear regression with multiple continuous predictors. VIF helps diagnose instability in coefficient estimates.
Does not apply to: single-predictor regression (there is no "other predictor" to regress against), pure prediction models where coefficient interpretation does not matter, or tree-based and neural network models that handle collinearity differently.
Limitation: VIF only detects pairwise or linear combinations among predictors. It does not tell you which variable to drop — domain knowledge decides that.

14.1.3 Visual Intuition

Picture a scatterplot with $X_{1}$ on the horizontal axis and $X_{2}$ on the vertical axis. Under multicollinearity, the points crowd tightly along a diagonal line — the two variables are nearly the same thing. Now imagine the regression plane for $Y$ trying to tilt independently along both $X_{1}$ and $X_{2}$ axes. Because the data lives on a narrow ridge, a tiny wiggle in the data can tip the plane dramatically. The coefficients bounce around because there is no data off the ridge to anchor them.

The takeaway: multicollinearity is a data geometry problem. The predictors live in a skinny subspace, so the model cannot pin down their separate slopes.

14.1.4 Pitfalls

Common traps with multicollinearity:

Dropping the wrong variable. VIF tells you there is a problem. It does not tell you which variable to keep. If you drop a causally important variable just because it is collinear, your model becomes biased. Use domain knowledge, not just VIF.
High VIF with a large sample. With enough data, even highly collinear predictors can produce stable estimates, because the standard errors shrink with $n$ . A VIF of 6 with $n = 10, 000$ may be tolerable. With $n = 30$ , it is panic time.
Ignoring multicollinearity in prediction-only models. If you only care about prediction accuracy and not coefficient interpretation, multicollinearity is less of a problem — the predictions may still be fine. But the coefficients will be uninterpretable, and that matters if anyone asks "which factor matters most?"

14.1.5 Recap and Bridge

Recap: Multicollinearity inflates coefficient variance when predictors are highly correlated. Detect it with VIF > 5. Fix it by dropping or combining variables. But dropping variables has a cost — you lose information. What if you could keep all variables but restrain the model from over-relying on any one of them? That is exactly what regularization does.

14.1.6 Real-World & Domain Connection

Multicollinearity appears everywhere in observational data. In economics, GDP, consumption, and investment all trend together — regressing one on the others produces sky-high VIFs. In health studies, BMI, waist circumference, and body fat percentage are all proxies for the same underlying construct. In marketing, ad spend on TV, radio, and social media often move together because budgets are set as a percentage of revenue. The analyst's job is to recognize when variables are redundant and to choose the right representative — or to use techniques like principal component regression that collapse collinear variables into uncorrelated components.

14.1.7 Student Questions and Answers

No specific student questions were raised on this topic during this session. The concept was presented as a quick recap from the previous regression discussion.

14.2 Overfitting and Regularization

Your model reports $R^{2} = 0.99$ . Should you celebrate or panic? An $R^{2}$ that close to 1.00 is not a trophy — it is a warning. The model has stopped learning patterns and started memorizing noise. It will embarrass itself on new data.

14.2.1 What Overfitting Looks Like

Imagine a student who memorizes the entire textbook — every comma, every page number, every example. On a practice test made from the textbook, they score 100%. Then the real exam asks a question they have never seen, in a slightly different way, and they freeze. They did not learn the concepts. They memorized the ink.

An overfitted model does the same thing. It gives you $R^{2} = 0.98$ or even $R^{2} = 1.00$ on training data. From a pure statistical standpoint, these numbers look fantastic. But the model has memorized the training points rather than learning the underlying relationship. It will fail on new data.

The professor's analogy: a colleague who is overactive — they do everything, maybe too much. You need mechanisms to pull them back so they perform at a sustainable level. Similarly, you pull an overfitted model back from 98% to maybe 80% or 70%, where it generalizes better.

Where the colleague analogy breaks: an overactive colleague eventually burns out on their own. An overfitted model does not self-correct — you must intervene with regularization.

14.2.2 How Regularization Works

In a simple linear regression $Y = w_{0} + w_{1} X$ , the goal is to find weights $w_{0}$ and $w_{1}$ that minimize the error — the gap between predicted $y^$ and actual $y$ . When the model overfits, you prevent it from taking whatever parameter values it wants. You impose a constraint on the weights themselves.

For a model with $n$ weights $w_{0}, w_{1}, \dots, w_{n}$ , the three core forms of regularization are:

L1 Regularization (Lasso):

w min i = 1 \sum m (y_{i} - y^_{i})^{2} subject to j = 0 \sum n ∣ w_{j} ∣ \leq c

The sum of absolute values of all weights must stay below a constant $c$ . This is the L1 norm constraint, written $∥ w ∥_{1} \leq c$ . Lasso tends to push some weights exactly to zero — so it does automatic feature selection. Irrelevant features drop out entirely.

L2 Regularization (Ridge):

w min i = 1 \sum m (y_{i} - y^_{i})^{2} subject to j = 0 \sum n w_{j}^{2} \leq c

The sum of squared weights must stay below $c$ . This is the L2 norm constraint, written $∥ w ∥_{2}^{2} \leq c$ . Ridge shrinks all weights toward zero but rarely pushes any all the way there. Every feature stays in the model, just with a milder influence.

Elastic Net:

Elastic Net combines both penalties in a single objective:

w min i = 1 \sum m (y_{i} - y^_{i})^{2} + λ_{1} j = 0 \sum n ∣ w_{j} ∣ + λ_{2} j = 0 \sum n w_{j}^{2}

This is the Lagrangian form of the constrained problems. Setting $λ_{1} = 0$ recovers Ridge. Setting $λ_{2} = 0$ recovers Lasso. Setting both to positive values gives a blended model that shrinks some weights to zero while shrinking others toward zero.

The equivalence between the Lagrangian form and the constraint form comes from Lagrange multipliers: $λ_{1}$ and $λ_{2}$ are the multipliers. $λ = 0$ means no constraint at all — plain linear regression. Larger $λ$ means tighter regularization — smaller weights, simpler model.

14.2.3 Symbol Registry

Symbol	Meaning	Type	Domain
$w_{j}$	weight (coefficient) for feature $j$	scalar	$R$
$m$	number of training examples	scalar	$N$
$n$	number of features (not counting intercept)	scalar	$N$
$λ$	regularization strength (Lagrange multiplier)	scalar	$λ \geq 0$
$λ_{1}$	L1 penalty weight (Lasso component)	scalar	$λ_{1} \geq 0$
$λ_{2}$	L2 penalty weight (Ridge component)	scalar	$λ_{2} \geq 0$
$y^_{i}$	predicted value for example $i$	scalar	$R$
$y_{i}$	actual (observed) value for example $i$	scalar	$R$
$c$	constraint bound on weight norm	scalar	$c \geq 0$
$∥ w ∥_{1}$	L1 norm: sum of absolute weights	scalar	$\geq 0$
$∥ w ∥_{2}^{2}$	squared L2 norm: sum of squared weights	scalar	$\geq 0$

14.2.4 Mathematical Foundation — Lagrange Multipliers

The constrained optimization forms — "minimize error such that sum of weights $\leq c$ " — are solved using Lagrange's method of multipliers. The $λ$ in the Elastic Net objective is the Lagrange multiplier.

Here is how the transformation works. Start with the constrained Ridge problem:

w min \sum (y_{i} - y^_{i})^{2} s.t. \sum w_{j}^{2} \leq c

Introduce the Lagrange multiplier $λ \geq 0$ and form the Lagrangian:

L (w, λ) = \sum (y_{i} - y^_{i})^{2} + λ (\sum w_{j}^{2} - c)

For a fixed $λ$ , minimizing $L$ with respect to $w$ is equivalent to:

w min \sum (y_{i} - y^_{i})^{2} + λ \sum w_{j}^{2}

The constant $- λ c$ drops out because it does not depend on $w$ . This is the familiar Ridge objective. The Lagrange multiplier $λ$ controls how hard the constraint bites: $λ = 0$ means no constraint (plain linear regression); larger $λ$ means tighter regularization, smaller weights, simpler model.

This is the same Lagrange multiplier technique you may have seen in mathematical foundations — applied here to penalize model complexity rather than to enforce an equality constraint.

14.2.5 Visual Intuition — Why L1 Gives Zeros and L2 Does Not

Picture a 2D weight space with axes $w_{1}$ and $w_{2}$ . The unregularized solution sits at the bottom of the error bowl — the $(w^_{1}, w^_{2})$ that minimizes squared error.

Now impose the L1 constraint $∥ w ∥_{1} \leq c$ . This is a diamond centered at the origin, with vertices on the axes. The L2 constraint $∥ w ∥_{2}^{2} \leq c$ is a circle centered at the origin.

The regularized solution is where the error contours first touch the constraint region as you expand outward from the unconstrained optimum. With the diamond (L1), the contours hit the sharp corner of the diamond — a point where one weight is exactly zero. With the circle (L2), the contours hit a smooth edge — both weights shrink but neither lands exactly at zero.

This geometric picture is why Lasso produces sparse solutions and Ridge does not. The diamond has corners on the axes; the circle does not.

14.2.6 Worked Example — L1 vs. L2 on Two Weights

Suppose you fit a model and get unregularized weights $w^_{1} = 3.0$ and $w^_{2} = 4.0$ . Now apply regularization.

L2 (Ridge) with $λ = 0.1$ :

The Ridge solution shrinks both weights proportionally. The new weights are roughly:

w_{1}^{ridge} \approx \frac{3.0}{1 + 0.1} = 2.73, w_{2}^{ridge} \approx \frac{4.0}{1 + 0.1} = 3.64

Both shrink by about 9%. Neither hits zero.

L1 (Lasso) with $λ = 0.5$ :

The Lasso solution applies a flat shrinkage: subtract $λ$ from each absolute weight, then clip negatives to zero:

w_{1}^{lasso} = max (0, 3.0 - 0.5) = 2.5, w_{2}^{lasso} = max (0, 4.0 - 0.5) = 3.5

Both survive because $λ$ is smaller than either weight. If $λ = 3.5$ , then $w_{1}^{lasso} = 0$ — feature 1 is eliminated.

Sense-check: In the exact Lasso solution, the shrinkage is not uniform like this soft-thresholding sketch — but the key behavior is correct: Lasso can zero out weights; Ridge cannot. With Ridge, a weight only hits zero if the unregularized weight was already zero.

14.2.7 Comparison — Lasso vs. Ridge vs. Elastic Net

Dimension	Lasso (L1)	Ridge (L2)	Elastic Net
Penalty	$λ \sum ∥ w_{j} ∥$	$λ \sum w_{j}^{2}$	$λ_{1} \sum ∥ w_{j} ∥ + λ_{2} \sum w_{j}^{2}$
Sparsity	Yes — drives some weights to exactly zero	No — shrinks all weights toward zero	Yes, but less aggressive than pure Lasso
Feature selection	Automatic	Manual (keep all)	Blended
Correlated features	Picks one, drops others	Shrinks both together	Balances both
When to use	You believe most features are irrelevant	You believe all features contribute somewhat	You are unsure — let the data decide

One-line decision rule: If you need sparsity (automatic feature selection), use Lasso. If all features are plausibly useful, use Ridge. If you are uncertain, start with Elastic Net and tune $λ_{1}$ and $λ_{2}$ .

14.2.8 Practical Strategy — When to Regularize

There is no single best practice. Two common approaches:

Approach 1 — Reactive: Build the model without regularization. Check performance on a validation set. If it overfits, add regularization and retrain.

Approach 2 — Proactive (recommended by the professor): Start with Elastic Net from the beginning. Set $λ_{1} = 0$ to check Ridge behavior. Set $λ_{2} = 0$ to check Lasso behavior. Tune both for intermediate trade-offs. This way the regularized model is already built in, and you explore the space systematically rather than retrofitting later.

Training time concern: In academic settings, training time is typically one to three minutes. In industry, complex data can push this higher — which is why the proactive approach avoids costly re-training from scratch.

14.2.9 Assumptions and Scope

Scope: When regularization helps — and when it hurts.

Helps when: you have many features relative to the number of observations ( $n ≫ m$ ), your features are correlated (multicollinearity), or you suspect many features are noise. Regularization reduces variance at the cost of some bias.
Hurts when: you have very few features, all of them are known to be causally important, and you have abundant data. In that case, regularization adds bias without meaningfully reducing variance.
Key assumption: the error term $\sum (y_{i} - y^_{i})^{2}$ is still the right loss function for your problem. If your data has heavy outliers, consider a strong loss instead.
$λ$ must be tuned: you cannot guess $λ$ . Use cross-validation to find the value that minimizes validation error. A $λ$ that is too large underfits; too small does not regularize enough.

14.2.10 Pitfalls

Common traps with regularization:

Regularizing the intercept $w_{0}$ . The intercept controls the baseline level — penalizing it makes no sense. Always exclude $w_{0}$ from the penalty term. Most libraries do this by default; if you implement it yourself, remember to skip the intercept.
Not standardizing features before regularizing. L1 and L2 penalties treat all weights equally. If one feature is measured in dollars (range 0–1,000,000) and another in proportions (range 0–1), the penalty hits the dollar feature much harder — not because it is less important, but because its scale is larger. Standardize all features to mean 0, variance 1 before regularizing.
Trusting $R^{2}$ on training data after regularization. Regularization intentionally degrades training fit to improve generalization. A drop in training $R^{2}$ from 0.98 to 0.80 is expected and good — check validation $R^{2}$ instead.
Using Lasso when all features are genuinely important. If every feature has a real, nonzero effect on $Y$ , Lasso will incorrectly zero some of them out. Use Ridge or Elastic Net with a small $λ_{1}$ .

14.2.11 Recap and Bridge

Recap: Overfitting is memorization, not learning. Regularization — L1 (Lasso), L2 (Ridge), or Elastic Net — restrains the model by penalizing large weights. Lasso zeros out irrelevant features; Ridge shrinks all features; Elastic Net blends both. The Lagrange multiplier $λ$ controls the trade-off between fit and simplicity. Now, before we move to time series, the professor pauses to give exam guidance on the correlation and regression module — exactly what to expect and how to prepare.

14.2.12 Real-World & Domain Connection

Regularization is not just an academic trick. In genomics, researchers fit models with thousands of gene expression features on only a few hundred patient samples — Lasso is the standard tool because most genes are irrelevant to any given disease. In finance, Ridge regression stabilizes portfolio optimization when asset returns are highly correlated. In natural language processing, Elastic Net helps select relevant words from massive vocabularies for text classification. The common thread: whenever you have more candidate features than you can trust, regularization keeps the model honest.

14.2.13 Student Questions and Answers

Q: Should I start with Lasso or Ridge? What guides the decision?

A: Several students asked variations of this question. The answer depends on sparsity. If you believe most features are irrelevant and want automatic feature selection, use Lasso — it drives irrelevant weights to exactly zero. If you believe all features contribute something and you just want to prevent extreme coefficients, use Ridge. The most flexible strategy: start with Elastic Net. Set $λ_{1} = 0$ to recover Ridge. Set $λ_{2} = 0$ to recover Lasso. Tune both together for the blended model. This lets you explore all options from one model structure.

Q: Does not training twice — once without regularization, then again with — take longer?

A: In academic work, training times are usually one to three minutes at most. In industry, data complexity can push this higher, which is precisely why the proactive approach (building with regularization from the start) is pragmatic — you avoid re-training from scratch. The professor noted that in academic settings, models rarely take more than three minutes to train, but in industry, complex real-world data can require substantially longer.

14.3 Exam Guidance — Correlation and Regression Recap

14.3.1 What to Expect for the Regression Module

Exam note: The correlation and regression module (pre-mid-semester) carries roughly 20% weight in the final exam. The post-mid-semester material — hypothesis testing and time series — carries roughly 80%. Allocate your study time accordingly.

The regression and correlation module is structured around a few predictable numerical problem types:

Covariance and correlation for a given dataset. Compute both from the data. Draw conclusions — compare what covariance tells you (direction and units of the relationship) versus what the correlation coefficient tells you (direction and strength, normalized to $[- 1, 1]$ ). Comment on the strength and direction of the relationship.
Simple linear regression. Given data, fit the line $Y = w_{0} + w_{1} X$ . Use it to predict $Y$ when $X$ is given a specific value (for example, "find $Y$ when $X = 20$ "). Comment on the trend — is it directly proportional ( $w_{1} > 0$ ), inversely proportional ( $w_{1} < 0$ ), linear?
$R^{2}$ interpretation. You may be asked to compute $R^{2}$ from the fitted model and then comment:
Exam note: Two canonical $R^{2}$ values to know cold:
- $R^{2} = 0.2$ : The model performance is very poor. The likely reason: the data has high nonlinearity but you forced a linear model onto it. A linear model cannot capture a curved pattern, so $R^{2}$ is low.
- $R^{2} = 0.8$ : The data is predominantly linear. The linear model captures the pattern well.
Linear vs. nonlinear regression decision. Given a dataset, how do you decide whether to use linear regression or nonlinear regression? The thought process: examine the data pattern visually (scatterplot), try a linear fit, check $R^{2}$ , and assess whether the residuals show systematic curvature. If residuals follow a U-shape or inverted-U, the relationship is nonlinear.

Exam note: For inference-type questions ("Comment on the trend," "Which model is better and why?", "What does this $R^{2}$ value tell you?"), write out all assumptions explicitly. Show every computational step. Partial credit depends on visible reasoning.

14.3.2 Study Strategy Tips

The consolidated formula sheet and problem set will be shared before the exam.
A dedicated revision and summary session will be held to go through problem-solving strategies and exam preparation planning.
Weight distribution reminder: ~80% post-mid-semester, ~20% pre-mid-semester (correlation and regression).
Expect both numerical problems and inference/commentary questions that test your conceptual understanding.

14.4 Time Series vs. Regression Models

A dataset has a timestamp column. Does that automatically make it a time series problem? No — and treating everything with a date column as a time series is one of the most common modeling mistakes. The real question is: does yesterday's value influence today's?

14.4.1 The Core Distinction

Imagine two filing cabinets. In the first cabinet, each folder is independent — you can pull them out in any order and the contents do not depend on what you pulled before. That is regression: data points are independent, and order does not matter.

In the second cabinet, each folder contains a note that says "see the previous folder for context." You must read them in order because each one builds on the last. That is time series: the value at time $t$ depends on values at earlier times. Shuffle the order and the whole thing becomes meaningless.

Where the analogy breaks: in a real filing cabinet, you could still read a folder in isolation and get some meaning. In a time series, $Y_{t}$ literally depends on $Y_{t - 1}, Y_{t - 2}, \dots$ — the dependency is mathematical, not just narrative.

Regression models are static. You model $Y = f (X)$ without any notion of sequence or time order. The data points are assumed to be independent of each other.

Time series models are dynamic. The data comes with a time index — $t = 1, 2, 3, \dots$ — and the value at time $t$ depends on values at earlier times. The order is part of the information. You cannot shuffle time series data without destroying the signal.

14.4.2 When the Distinction Gets Tricky

Almost all modern sensor data arrives with a timestamp — health monitors, BP readings, step counts, X-ray timestamps. The mere presence of a time column does not automatically make it a time series problem. The key diagnostic question:

"Does the output truly depend on time, or is time just a label?"

If yesterday's BP influences today's BP, it is a time series problem — the time dependency carries signal. If time is just a tag and the output depends only on other features (age, exercise habits, weight), then regression may work fine. The validation test: ask whether $Y_{t}$ at minute $t$ really depends on $t$ . If yes → time series model. If no → regression may be appropriate.

14.4.3 The Random Split Problem

In standard ML, you split data into training and testing sets randomly. Random shuffling prevents bias — it ensures both sets see a representative mix of the data.

In time series, you cannot shuffle randomly. The sequence order must be preserved. If you are predicting sales for the 8th month, the model must see months 1 through 7 in order during training. A random split would leak future information into the training set — the model would cheat by seeing data from months 9–12 before predicting month 8.

The standard time series split: use the first 80% of the timeline for training, the last 20% for testing. The split is chronological, not random. This is why time series needs its own family of models — standard ML models assume independent, identically distributed (i.i.d.) data, and that assumption does not hold for ordered sequences.

Worked Example: Chronological vs. Random Split

Suppose you have 12 months of monthly sales data: Jan, Feb, ..., Dec. You want to build a model to forecast next month's sales.

Wrong (random split): Shuffle all 12 months. Train on 9 random months, test on 3 random months. The model might train on November sales and then be tested on March sales. This is cheating — the model saw the future during training.
Correct (chronological split): Train on Jan–Sep (first 75%). Test on Oct–Dec (last 25%). The model sees only past data during training and is tested on genuinely future data. This mimics real forecasting, where you never have access to the future.

Sense-check: If your model had access to future data during training, its test performance would look artificially good — but it would fail catastrophically in production, where the future is genuinely unknown.

14.4.4 Comparison — Time Series vs. Regression

Dimension	Regression	Time Series
Data assumption	i.i.d. (independent, identically distributed)	Autocorrelated (each point depends on past)
Order matters?	No — shuffle freely	Yes — order is the signal
Train/test split	Random	Chronological (time-based)
Core question	"How does $X$ affect $Y$ ?"	"How does the past affect the future?"
Example	Predict house price from square footage	Predict tomorrow's stock price from recent history

When to pick which: If shuffling your data would destroy meaningful patterns, use time series models. If shuffling changes nothing, regression is fine.

14.4.5 Beyond ML — Statistical and Deep Learning Models

Because ML models assume random shuffling, time series problems historically relied on statistical models — ARIMA, exponential smoothing, and their variants. These were developed decades ago (ARIMA dates to the 1970s) and remain widely used in business forecasting because they are interpretable, computationally cheap, and work well on small-to-medium datasets.

When statistical models are not sufficient — nonlinear patterns, long-range dependencies, high-dimensional data — you move to deep learning models, specifically sequential models like RNNs (Recurrent Neural Networks) and LSTMs (Long Short-Term Memory networks). These are designed to handle ordered sequences and can capture complex temporal dependencies that linear statistical models miss.

The same sequential nature appears outside time series. In language translation, the input sentence has a word order you cannot shuffle. "The cat sat" is not the same as "sat cat The." RNNs and LSTMs are used for both time series forecasting and NLP — both involve sequences where order must be preserved. The underlying mathematical structure is the same; only the domain differs.

14.4.6 Visual Intuition

Plot your data with time on the horizontal axis and $Y_{t}$ on the vertical axis. A regression dataset looks like a cloud of points with no obvious left-to-right pattern — the value at $t = 5$ tells you nothing about $t = 6$ . A time series has visible structure: points are connected by lines, trends rise or fall, seasonal bumps repeat. The eye can see the autocorrelation.

Now imagine shuffling both plots. The regression cloud looks the same — independence means shuffling is a no-op. The time series plot becomes a scrambled mess — all the trend and seasonality vanish, confirming that order was carrying real information.

14.4.7 Pitfalls

Common traps when deciding between regression and time series:

"It has a timestamp, so it must be time series." The presence of a time column is not enough. Validate whether $Y_{t}$ actually depends on $t$ . A dataset of patient measurements where each row is a different patient with a measurement date is not a time series — it is cross-sectional data with a date stamp.
Randomly shuffling time series data for cross-validation. This is the cardinal sin. You will get inflated performance metrics because the model learns from future data that would not be available in production.
Using regression when there is weak but real time dependence. Even a small autocorrelation ( $Y_{t}$ correlates with $Y_{t - 1}$ at 0.3) means the i.i.d. assumption is violated. Standard errors and p-values from a regression will be wrong.
Assuming time series models are always better. If there is genuinely no time dependence — verified by checking autocorrelation plots — a regression model is simpler, faster, and equally accurate. Do not overcomplicate.

14.4.8 Recap and Bridge

Recap: Time series data has order-dependent structure; regression data does not. The diagnostic question is "does the past influence the present?" If yes, preserve chronological order and use time series methods. If no, regression is fine. Now that we know time series is its own category, the next question is: what are its building blocks? Every time series can be decomposed into trend, seasonality, cyclicality, and irregularity.

14.4.9 Real-World & Domain Connection

The regression-vs-time-series decision is not academic. A hospital predicting patient readmission risk from static features (age, diagnosis codes, lab values) can use regression — each patient is independent. But the same hospital predicting hourly ICU bed occupancy must use time series — occupancy at 3 PM depends on occupancy at 2 PM. In retail, predicting which customers will churn (regression) is fundamentally different from predicting next week's total sales (time series). The decision drives everything downstream: model choice, validation strategy, and deployment architecture.

14.4.10 Student Questions and Answers

Q: If the data depends on time as well as other features like age and exercise habits, is it time series or regression?

A: If the output depends on time — that is, $Y_{t}$ is influenced by $Y_{t - 1}$ or the time index $t$ — it is a time series problem. The presence of additional features (age, exercise) does not change this classification. For example, BP data that depends on time, age, and exercise habits is still a time series analysis. The time dependency is part of what drives the output, and you must model it as such.

Q: If there is no trend and no seasonality in the data, does a time series model default to behaving like regression?

A: That is exactly the right intuition. The mere presence of a time column does not make data a time series. You validate whether the value at time $t$ genuinely depends on the time step. If there is no trend, no seasonality, and no autocorrelation, a regression approach may work perfectly well — and it will be simpler. The time series model would effectively reduce to something very close to a regression in that case.

Q: In ML, we treat months as categorical with no inherent order — January is not "greater than" February. So why does time series treat months as ordered?

A: In time series, the sequence matters, not the magnitude comparison. January is not numerically larger than February, but January comes before February. The order — what happens when — is what the model captures. In a sales presentation, you would never plot months in random order; the timeline is the whole point. Time series preserves that timeline. The model learns that values in adjacent months are related, not because one is bigger, but because they are neighbors in time.

14.5 Components of a Time Series

What is a time series actually made of? If you look at a stock chart, you see a general upward drift, regular bumps every quarter, wavy business-cycle swings, and random daily jitter. These are not one thing — they are four separate components layered on top of each other. Decomposing them is the first step in any time series analysis.

14.5.1 Four Core Components

Think of a time series like a dish at a restaurant. You taste the final product — but a chef can break it down into its ingredients: the base stock (trend), the recurring spice blend (seasonality), the variable fresh herbs (cyclicality), and the accidental extra pinch of salt (irregularity). Each ingredient contributes differently, and you need to identify them separately to understand, or replicate, the dish.

A time series decomposition does the same thing: it separates the observed series $Y_{t}$ into up to four underlying components. Not every series has all four, and the ones that are absent simply contribute nothing (zero in the additive model, one in the multiplicative model).

Every time series can be decomposed into up to four components:

Component	Notation	Description	Period
Trend	$T_{t}$	Long-term direction — increasing, decreasing, or flat — over an extended period	Decades, years
Seasonality	$S_{t}$	Regular, fixed-period repeating pattern — defined by the analyst	Known, fixed (e.g., every summer, every December)
Cyclicality	$C_{t}$	Wave-like patterns with no fixed period — observed from data	Variable, observed
Irregularity	$I_{t}$	Random noise, shocks, one-off events — everything else	None

14.5.2 Trend

Trend is the long-run movement — the slow, persistent direction of the series after stripping away everything else. Population growth over 50 years shows an upward trend. The stock market over decades shows an upward trend. When governments and planning commissions forecast for the next decade or century, they look at the trend component and deliberately ignore month-to-month wiggles.

A trend does not have to be linear. It can be:

Linear: $T_{t} = β_{0} + β_{1} t$ — constant growth per unit time.
Quadratic: $T_{t} = β_{0} + β_{1} t + β_{2} t^{2}$ — accelerating or decelerating growth.
Exponential: $T_{t} = β_{0} e^{β_{1} t}$ — growth proportional to current level.
Autoregressive: $T_{t} = ϕ T_{t - 1}$ — the trend itself evolves based on its own past.

Different trend shapes require different modeling approaches. The choice of trend model determines how far you can reasonably forecast.

14.5.3 Seasonality

Seasonality is a pattern that repeats over a known, fixed period. The period is set by you, the analyst — you define the calendar window and check whether the pattern repeats:

Refrigerator sales spike every summer. The season is "summer" — a fixed calendar window.
Gold purchases rise near Diwali every year. The season is "Diwali" — a fixed festival date.
Retail sales in the US jump every December. The season is "December" — a fixed month.

Seasonality is what you define. You do not discover it from the data — you hypothesize it ("sales probably spike in December") and then verify. This is the key difference from cyclicality.

14.5.4 Cyclicality

Cyclicality looks like seasonality but the period is not fixed. You observe it from the data rather than defining it upfront:

Every two months, something happens. You notice this by looking at the data, not by declaring "every 60 days."
Business cycles — every 7–8 years the economy slows down — are cyclical, not seasonal. The exact timing varies.
Flu season has a rough periodicity, but the onset and duration shift each year.

The difference in one line: with seasonality, you say "every October." With cyclicality, you observe "it seems to repeat roughly every two months, but the spacing is not exact."

14.5.5 Comparison — Seasonality vs. Cyclicality

Dimension	Seasonality	Cyclicality
Period	Fixed, known, pre-defined by analyst	Variable, observed from data
How identified	You declare the window and verify	You discover the pattern from the data
Typical length	Within a year (monthly, quarterly, weekly)	Multi-year (business cycles, 7–8 years)
Predictability	Highly predictable — same month next year	Less predictable — timing varies
Example	Ice cream sales every July	Economic recessions every 7–10 years

When to classify as which: If you can write the period on a calendar (every December, every Monday), it is seasonality. If you can only describe the period roughly ("roughly every two months"), it is cyclicality.

14.5.6 Irregularity (Random Component)

Irregularity is the residual — random shocks, noise, one-off events that do not fit any pattern:

COVID-19 lockdowns drove pollution levels to near-zero for two years. That is an irregular event. Including those two years when modeling the long-term pollution trend would produce forecasts of unrealistically clean air.
A sudden market crash from a geopolitical event.
A supply chain disruption from a natural disaster.

Scope: The analyst must decide — include or exclude?

The pollution example is instructive. COVID produced near-zero readings that were not part of the underlying pollution-generating process. Excluding them is correct — they would distort the trend. But in agriculture, weather-driven yield variation is inherent randomness that must be modeled because it is part of the system every season. The decision depends on domain knowledge:

Exclude when the irregularity is a rare, external shock that does not reflect the underlying process (COVID pollution, one-time factory shutdown).
Include when the irregularity is structural and ongoing — it is not really "irregular" in the statistical sense, just inherently variable (agricultural yields, daily rainfall).

14.5.7 Visual Intuition

Plot a time series and mentally layer the four components:

Trend is the smooth, slowly-changing line you would draw through the middle, ignoring all bumps. Think of a 50-year moving average.
Seasonality is the regular heartbeat — identical bumps at identical intervals. Every July peaks; every January troughs.
Cyclicality is the slower, irregular swell — like ocean waves that arrive at roughly consistent but not exact intervals.
Irregularity is the static, the fuzz, the unexplained jitter — what remains after you subtract trend, seasonality, and cyclicality.

The takeaway: a single data point $Y_{t}$ is the combined effect of all four forces. Decomposition untangles them so you can model each one with the right tool.

14.5.8 Pitfalls

Common traps with time series components:

Confusing seasonality with cyclicality. If you can put the period on a calendar, it is seasonal. If not, it is cyclical. Students often call everything "seasonal" — be precise.
Removing the trend when you should not. De-trending is standard for stationary models like ARIMA, but if your forecasting goal is long-term (decades), the trend IS the signal. Do not remove it.
Treating all irregularity as noise to discard. Some irregularity is structural (agriculture, finance). Removing it removes genuine uncertainty from your forecast, giving you overconfident prediction intervals.
Forgetting that components can interact. In a multiplicative model, a 10% seasonal swing when the trend is at 100 is a swing of 10 units. When the trend is at 1,000, the same seasonal percentage is a swing of 100 units. The components are not always independent.

14.5.9 Recap and Bridge

Recap: Every time series decomposes into trend (long-run direction), seasonality (fixed-period cycles), cyclicality (variable-period waves), and irregularity (random noise). The decomposition guides model choice: trend models for long-range planning, seasonal models for inventory management, cyclical models for business cycle analysis. Now the question is: once you have the components, how do they combine to produce the observed value? They can add or multiply — and that choice matters.

14.5.10 Real-World & Domain Connection

Time series decomposition is not just a textbook exercise. The US Bureau of Labor Statistics decomposes employment data to separate the long-term structural trend from seasonal hiring patterns — so policymakers can see whether a December employment dip is just the usual post-holiday layoff cycle or a genuine economic downturn. Retailers decompose sales to isolate Black Friday spikes (seasonality) from multi-year brand growth (trend). Epidemiologists decompose disease incidence to distinguish the expected flu season from an anomalous outbreak (irregularity). In every case, the decomposition answers the same question: is this movement normal, or is something new happening?

14.5.11 Student Questions and Answers

Q: When working with second-by-second sensor data, how do you aggregate it for meaningful analysis?

A: Compress it. Second-by-second data has enormous volume but often little signal at that granularity. Aggregate to minute-level or 5-minute-level. The aggregation method depends on the use case. Use the average if you need central tendency. Use the maximum if you need peak detection. Use the minimum for lower-bound analysis. The choice is one of the central tendency measures — mean, median, max, min — driven by what question you are answering.

Q: Do all four components always appear in every time series?

A: No. Not every time series has all four. A very disciplined, well-behaved dataset may have only trend and seasonality with no cyclicality and no irregularity. The components that are absent simply drop out — in the additive model, the term is zero; in the multiplicative model, the term is one. The model does not force absent components to exist.

Q: When we removed the COVID zero-pollution period from the model, that was treating it as irregularity. But what if irregularity is inherent to the domain — like agriculture, where weather causes irregular crop yields every season?

A: This is an important distinction. In agriculture, irregularity cannot be removed because it is part of the system. Weather-driven yield variation is inherent randomness that your model must capture — it represents genuine uncertainty about future yields. The COVID pollution anomaly was a rare, external shock that did not reflect the underlying pollution-generating process — you can exclude it. The agricultural irregularity is ongoing and structural — you must model it. The decision of whether to include or exclude an irregular component depends on domain knowledge and the nature of the irregularity: is it a one-off shock, or is it inherent variability?

Q: Can an irregular event become a regular pattern over time?

A: Yes — several students asked about this with the mask-wearing example. What starts as an irregular shock can become embedded as a structural change in the data. During COVID, mask-wearing was an irregular response. If mask-wearing persists after COVID, the time series model would eventually absorb this as part of the new normal — it could show up as a shift in the trend level or even as a new seasonal pattern (e.g., higher mask sales every flu season). Time series models adapt; they do not treat "irregular" as permanently irregular.

14.6 Additive and Multiplicative Models

You have the four components — trend, seasonality, cyclicality, irregularity. How do they combine to produce the number you actually observe? They either add together or multiply together. The choice between additive and multiplicative is not arbitrary — it depends on whether the seasonal swings stay the same size or grow with the trend.

14.6.1 Two Ways to Combine the Components

Imagine a small shop and a large supermarket chain. In December, both see a holiday sales bump. The small shop's December sales jump by 10,000 rupees — a fixed amount. The supermarket chain's December sales jump by 30% — a percentage that scales with their overall volume.

The additive model describes the small shop: the seasonal bump is a fixed number of rupees, independent of how big the business is. The multiplicative model describes the supermarket: the seasonal bump is a percentage, so it grows as the business grows.

Where the analogy breaks: in reality, components do not have to all follow the same rule. You could have additive seasonality with multiplicative irregularity — but the standard models keep it simple with a single combination rule.

Additive Model:

Y_{t} = T_{t} + S_{t} + C_{t} + I_{t}

The observed value is the sum of the four components. Each component contributes independently — a change in seasonality does not affect the trend. Use this model when the magnitude of seasonal swings is roughly constant over time. The peaks and troughs have similar amplitude regardless of the trend level.

Multiplicative Model:

Y_{t} = T_{t} \times S_{t} \times C_{t} \times I_{t}

The observed value is the product of the four components. The seasonal effect scales with the trend — when the trend is high, the seasonal swings are larger in absolute terms. Use this model when the amplitude grows (or shrinks) over time.

Log Transformation Trick:

Taking the natural log converts the multiplicative model into an additive one:

lo g (Y_{t}) = lo g (T_{t}) + lo g (S_{t}) + lo g (C_{t}) + lo g (I_{t})

This is mathematically convenient — you can use additive-model techniques on the log-transformed data, then exponentiate to get back to the original scale. Many software packages do this internally.

14.6.2 Comparison — Additive vs. Multiplicative

Dimension	Additive	Multiplicative
Formula	$Y_{t} = T_{t} + S_{t} + C_{t} + I_{t}$	$Y_{t} = T_{t} \times S_{t} \times C_{t} \times I_{t}$
Seasonal amplitude	Constant over time	Grows or shrinks with trend
Component independence	Components do not interact	Components scale each other
Absent component	Contributes 0	Contributes 1
Log transform	Not needed	$lo g (Y_{t})$ makes it additive
Visual cue	Peaks and troughs are equally tall	Peaks get taller as the series rises

When to pick which: Plot your time series. If the seasonal swings are roughly the same height from start to finish, use additive. If the swings get larger as the overall level rises (a common pattern in growing businesses), use multiplicative.

14.6.3 Worked Example — Same Components, Two Models

Suppose at time $t$ , the components are: $T_{t} = 100$ , $S_{t} = 20$ , and $C_{t} = I_{t} = 1$ (no cyclicality or irregularity).

Additive: $Y_{t} = 100 + 20 = 120$ . The seasonal effect adds 20 units, period.

Multiplicative: $Y_{t} = 100 \times 20 \times 1 \times 1 = 2000$ . Wait — that seems wrong. In a multiplicative model, the seasonal component $S_{t}$ is expressed as a factor around 1, not as a raw number. A 20% seasonal bump would be $S_{t} = 1.20$ , giving $Y_{t} = 100 \times 1.20 = 120$ — matching the additive result for this case.

But now let the trend grow: at a later time, $T_{t} = 200$ , same seasonal factor $S_{t} = 1.20$ .

Additive (if we incorrectly used it): $Y_{t} = 200 + 20 = 220$ . The seasonal bump is still 20 units.
Multiplicative: $Y_{t} = 200 \times 1.20 = 240$ . The seasonal bump is now 40 units — it grew with the trend.

Sense-check: In the additive model, the seasonal swing in absolute units never changes, even as the business triples in size. That is unrealistic for most growing businesses. The multiplicative model captures the intuition that a bigger base means bigger absolute swings.

14.6.4 When Components Are Absent

If a component is missing: in the additive model, its term is $0$ (no contribution). In the multiplicative model, its term is $1$ (multiplying by 1 changes nothing). The other components are unaffected in either case.

14.6.5 Visual Intuition

Plot two time series side by side:

Additive series: The line oscillates around the trend with constant amplitude. The distance from peak to trough is the same at the beginning of the series (when the trend is low) and at the end (when the trend is high). It looks like a straight pipe with waves inside.
Multiplicative series: The line fans out. Near the start, the oscillations are small. Near the end, they are large — the peaks shoot higher and the troughs dip deeper. It looks like a megaphone opening up.

The takeaway: if your time series plot shows a fanning or megaphone shape, you need the multiplicative model. If the wave height is constant, additive is fine.

14.6.6 Assumptions and Scope

Scope: When each model applies and when it breaks.

Additive model assumes: components are independent; seasonal variation is constant regardless of trend level. Breaks when the data fans out — you will underpredict peaks and overpredict troughs at high trend levels.
Multiplicative model assumes: components scale proportionally; seasonal variation is a percentage of the trend level. Breaks when there is no trend (division by a trend near zero amplifies noise) or when the relationship is more complex than pure multiplication.
Log transform caveat: $lo g (Y_{t})$ only works when $Y_{t} > 0$ for all $t$ . If your series has zeros or negative values, you cannot take the log — use a different transformation or stick with the pure multiplicative decomposition.

14.6.7 Pitfalls

Common traps with additive and multiplicative models:

Using additive when the data fans out. If your plot shows growing amplitude, an additive model will systematically underpredict peaks at later times. Check for fanning before choosing.
Forgetting that multiplicative components are factors around 1. A seasonal "bump" of 20% is $S_{t} = 1.20$ , not $S_{t} = 20$ . If you plug $S_{t} = 20$ into a multiplicative model, you get absurd numbers.
Taking logs of zero or negative values. The log trick is elegant, but $lo g (0)$ is undefined and $lo g (negative)$ is complex. If your data has zeros, add a small constant first — but know that this changes the interpretation.
Assuming the same model for all components. In advanced settings, you can mix: additive trend with multiplicative seasonality. The basic models covered here assume a uniform rule, but real-world implementations are more flexible.

14.6.8 Recap and Bridge

Recap: Additive models sum the components (

Y_{t} = T_{t} + S_{t} + C_{t} + I_{t}

) — constant seasonal amplitude. Multiplicative models multiply them (

Y_{t} = T_{t} \times S_{t} \times C_{t} \times I_{t}

) — seasonal amplitude scales with trend. The log transform converts multiplicative to additive. Now, with decomposition and combination rules in hand, we turn to the simplest forecasting method: the moving average.

14.6.9 Real-World & Domain Connection

The additive-vs-multiplicative choice is not academic. Airlines forecasting passenger numbers use multiplicative models because passenger volume grows over time and seasonal swings (summer travel peaks) grow proportionally. A small regional airline with 10,000 monthly passengers and a major carrier with 1,000,000 both see summer bumps — but the major carrier's bump is 100 times larger in absolute terms. Using an additive model for the major carrier would dramatically underpredict summer demand and overpredict winter demand. The multiplicative model captures the scaling correctly.

14.6.10 Student Questions and Answers

Q: When we build a time series model, do we always include all four components?

A: No. You decompose the data first and inspect each component. If irregularity is absent, it drops out (term = 0 in additive, term = 1 in multiplicative). If you are only interested in seasonality — say, for refrigerator sales planning — you may focus entirely on that component and ignore the long-term trend. The model choice follows both the decomposition analysis and the specific business question you are trying to answer.

14.7 Simple Moving Average (MA) Models

How do you smooth out random noise to see the underlying pattern? Take an average — but not a single average of all the data. Take an average of a small sliding window. As the window moves forward in time, the average moves with it, tracing a smoother version of the original series. That is the moving average.

14.7.1 The Intuition

You already know moving averages from stock charts. Those MA(3), MA(5), and MA(50) lines overlaid on price charts are moving averages. A 3-day moving average smooths out daily noise so you can see the short-term direction. A 50-day moving average reveals the medium-term trend. The longer the window, the smoother the line — and the slower it responds to new changes.

Think of it like looking at a jagged coastline from an airplane. At low altitude (short window), you see every inlet and rock. At high altitude (long window), the coastline smooths out — you see the big shape, not the details. Moving averages give you altitude control over your data.

A moving average takes the average of a sliding window of

k

consecutive values. The average "moves" as the window slides forward in time. The choice of window length

k

determines the trade-off: smaller

k

is more responsive to recent changes but noisier; larger

k

is smoother but slower to react.

14.7.2 Centered Moving Average

For model-fitting (not forecasting), we use the centered moving average, where the forecast is assigned to the center of the window:

3-year centered MA:

Y^_{t} = \frac{Y _{t - 1} + Y _{t} + Y _{t + 1}}{3}

The first and last years in the dataset get no forecast — they lack a neighbor on one side.

5-year centered MA:

Y^_{t} = \frac{Y _{t - 2} + Y _{t - 1} + Y _{t} + Y _{t + 1} + Y _{t + 2}}{5}

The first two and last two years get no forecast.

14.7.3 Worked Example — 3-Year Centered Moving Average

Consider annual production data:

Year	Actual Production ( $Y_{t}$ )	3-Year MA Forecast ( $Y^_{t}$ )
1995	21	— (no prior year)
1996	22	$(21 + 22 + 23) /3 = 22$
1997	23	$(22 + 23 + 24) /3 = 23$
1998	24	— (no following year)

1995 has no forecast because it lacks a prior year ( $Y_{1994}$ is not in the data). 1998 has no forecast because it lacks a following year ( $Y_{1999}$ ). Under a centered moving average scheme, only interior points with full windows on both sides get forecasts.

The calculation for 1996: $(21 + 22 + 23) /3 = 66/3 = 22$ . The forecast (22) happens to match the actual value (22) — this is a coincidence for this particular dataset, not a property of the method.

Sense-check: The forecast for 1997 is $(22 + 23 + 24) /3 = 69/3 = 23$ , also matching the actual. These matches indicate the data is very smooth with little irregularity — not that the method is perfect.

14.7.4 Model Selection — MA(3) vs. MA(5)

You compute forecasts for both window sizes and compare them against actual values using error. For each year where both models produce a forecast:

Compute absolute error: $∣ e_{t} ∣ = ∣ Y_{t} - Y^_{t} ∣$ for MA(3)
Compute absolute error: $∣ e_{t} ∣ = ∣ Y_{t} - Y^_{t} ∣$ for MA(5)
Sum the absolute errors across all comparable points or compute the mean

The model with the smaller total (or mean) absolute error is the better fit. This is analogous to a loss function — you minimize the gap between actual and predicted.

Pitfall: You cannot tell which window length is better just by looking at the graph — it can be visually misleading. A smoother line looks nicer but may miss real turning points. Always compute the error numerically.

14.7.5 Forecasting Future Values with MA

Once you have settled on a window length (say, $k = 3$ ), you use the model to forecast beyond the data. But here the logic shifts: for forecasting, you cannot use a centered window because the future values do not exist yet. Instead, you use a trailing moving average — the average of the most recent $k$ known values:

Y^_{t + 1} = \frac{Y _{t - 2} + Y _{t - 1} + Y _{t}}{3}

For example, to forecast 2005 using data that ends at 2004:

Y^_{2005} = \frac{Y _{2002} + Y _{2003} + Y _{2004}}{3}

This slides forward — always the most recent three known values — to produce the next forecast. This is the same principle stock analysts use: the 3-day MA forecast for tomorrow is the average of today, yesterday, and the day before.

Centered MA is for model-fitting (evaluating how well the model describes historical data). Trailing MA is for forecasting (predicting future values you have not seen). They use the same averaging idea but apply the window differently because forecasting cannot peek at the future.

14.7.6 Visual Intuition

Plot your original time series as a jagged line of connected points. Overlay the 3-year moving average as a smoother line that passes through the middle of the wiggles. Then overlay the 5-year moving average as an even smoother line with fewer wiggles.

The original series hits every peak and trough. MA(3) tones them down — the peaks are lower, the troughs are shallower. MA(5) goes further — it is nearly a straight line through the data. The takeaway: as $k$ increases, the MA line approaches the long-term trend and filters out progressively more short-term variation. The cost: it also responds more slowly when the trend genuinely changes direction.

14.7.7 Assumptions and Scope

Scope: When moving averages work and when they break.

Works best for: Series with a horizontal pattern (no strong trend) or series where you only need to smooth, not forecast far ahead. Short-range forecasts — one or two steps ahead.
Breaks when: The series has a strong trend — a moving average will systematically lag behind because it averages past values that are lower than current values. The series has strong seasonality — simple MA does not capture seasonal patterns.
Window length trade-off: Smaller $k$ tracks changes faster but is noisier. Larger $k$ is smoother but slower to react to real shifts. There is no universally optimal $k$ — it depends on the data's volatility and your tolerance for lag.
Even-period MAs require centering adjustments. A 4-period MA falls between two time points; you need a 2×4 MA (centered moving average of a moving average) to align it with actual time periods.

14.7.8 Pitfalls

Common traps with moving averages:

Using centered MA for forecasting. Centered MA uses future data ( $Y_{t + 1}$ ) that does not exist when you are making a real forecast. Always switch to trailing MA for out-of-sample predictions.
Choosing $k$ by eyeballing the plot. A smoother line always looks better, but it may be missing real turning points. Compute error metrics and let the data decide.
Forgetting that boundary points have no forecast. The first and last $⌊ k /2 ⌋$ points have no centered MA forecast. This is expected — do not try to force a forecast at the edges during model-fitting.
Applying MA to trending data without detrending. If the series trends upward, the MA will systematically underpredict because it averages older, lower values with newer, higher ones. Remove the trend first, or use a method designed for trending data.

14.7.9 Recap and Bridge

Recap: The moving average smooths a time series by averaging a sliding window of

k

values. Centered MA fits the model to historical data; trailing MA forecasts future values. Choose

k

by comparing error across candidates — smaller

k

is responsive, larger

k

is smooth. But what if the most recent data should matter more than older data in the window? That is where weighted moving averages and exponential smoothing come in.

14.7.10 Real-World & Domain Connection

Moving averages are everywhere in finance — every trading platform plots MA(50) and MA(200) lines, and traders watch for "golden crosses" (when a short MA crosses above a long MA) as buy signals. In manufacturing, quality control charts use moving averages to detect when a production process drifts out of specification — a sudden shift in the MA triggers an alarm. In epidemiology, 7-day moving averages of COVID case counts became the standard way to report trends during the pandemic because they smoothed out the weekend reporting lull. The principle is universal: average out the noise to see the signal.

14.7.11 Student Questions and Answers

Q: For the first entry (1995), how do we compute a 3-year moving average? There are no prior entries.

A: The first and last entries have no forecast under a centered moving average scheme — they lack the required neighbors on one side. This is expected. You fit the model to the interior points where full windows exist. The missing boundary forecasts do not affect model selection because you evaluate performance only on points where both the model forecast and the actual value exist.

Q: For the final year in the dataset, if we only have two data points from the end (with no year after), can we still compute a centered 3-year average?

A: Under a centered moving average, no — you need a full window of three around the center point. The last year lacks a subsequent year, so it gets no centered-MA forecast. Once the model is validated and you move to forecasting future values, the logic changes: you use a trailing window of the last three known values. For example, to forecast 2005, use 2002, 2003, and 2004. Model-fitting handles boundary gaps; forecasting uses only the most recent window.

Q: Is MA(3) or MA(5) better for a given dataset — can we tell from the graph?

A: Several students asked this. You cannot reliably tell just by looking at the graph — it can be visually misleading. A smoother line (MA(5)) always looks more appealing, but it may be missing real turning points that MA(3) captures. Compute the absolute error $∣ Y_{t} - Y^_{t} ∣$ for both models across all comparable points. Pick the model with the smaller total error. The error analysis is the definitive comparison.

Q: What about the "MA Sum" column — is that a labeling issue?

A: The column labeled "MA Sum" in some example tables is actually the 5-year moving sum (the numerator before dividing by 5), not a 3-year computation. It is a labeling inconsistency. When in doubt, verify by checking whether the numbers match a 3-year or 5-year window calculation: if the sum has five terms, it is the 5-year numerator.

Q: Can we use even-period moving averages like 2-year or 4-year?

A: Yes. Moving averages can use any window length — 2, 3, 4, 5, 10, 50. The same principle applies. In practice, window lengths match the relevant time unit: 3 days, 5 weeks, 10 months, or 50 days. The granularity of the data and the forecasting horizon drive the choice. Even-period MAs need a slight adjustment (centering) because the average of an even number of time points falls between two periods, not on one.

Q: What if the time series has multiple input features, not just a single $Y$ over time?

A: That is an advanced scenario beyond the scope of basic models. When multiple predictor variables accompany the time dimension, you need multivariate time series models (like VAR — Vector Autoregression). The models discussed here — MA, exponential smoothing, ARIMA — are univariate: one output variable over time. Multivariate models exist but are covered in more advanced courses.

14.8 Weighted Moving Average

Should last week's gold price and last year's gold price carry equal weight when forecasting tomorrow? Of course not. The simple moving average treats every value in the window as equally important. The weighted moving average lets you say: recent data matters more.

14.8.1 When Equal Weights Are Not Enough

Think of a weather forecast. Yesterday's temperature tells you a lot about today's. The temperature from 30 days ago tells you almost nothing. If you averaged them equally, you would wash out the useful recent signal with irrelevant old noise.

The weighted moving average fixes this. You assign higher weights to recent observations and lower weights to older ones. The further back in time, the less influence. This is the same intuition that drives exponential smoothing — but with weights you choose explicitly rather than an exponential decay formula.

For a 3-period weighted moving average:

Y^_{t + 1} = w_{1} Y_{t} + w_{2} Y_{t - 1} + w_{3} Y_{t - 2}

where the weights must sum to 1:

w_{1} + w_{2} + w_{3} = 1

and typically the most recent observation gets the largest weight: $w_{1} > w_{2} > w_{3}$ .

14.8.2 Worked Example

Suppose gold prices for the last three days are: today = 7200, yesterday = 7150, two days ago = 7100 (in rupees per 10 grams).

Simple 3-day MA: Each day gets weight $\frac{1}{3}$ :

Y^_{tomorrow} = \frac{7200 + 7150 + 7100}{3} = 7150

Weighted 3-day MA with weights $(0.7, 0.2, 0.1)$ :

Y^_{tomorrow} = 0.7 \times 7200 + 0.2 \times 7150 + 0.1 \times 7100

= 5040 + 1430 + 710 = 7180

The weighted forecast (7180) is higher than the simple MA (7150) because it puts 70% of the weight on today's higher price. It is more responsive to the recent upward movement.

Sense-check: If today's price had dropped instead, the weighted MA would drop faster than the simple MA. Higher weight on recent data means faster reaction to changes — for better or worse.

14.8.3 Comparison — Simple MA vs. Weighted MA vs. Exponential Smoothing

Dimension	Simple MA	Weighted MA	Exponential Smoothing
Weight pattern	Equal for all $k$ values	Chosen by analyst; must sum to 1	Exponential decay: $α, α (1 - α), α (1 - α)^{2}, \dots$
Number of parameters	1 ( $k$ )	$k$ (all weights)	1 ( $α$ )
Flexibility	Low	High (but more to tune)	Moderate (one knob)
Responsiveness	Depends on $k$	Depends on weight distribution	Depends on $α$
When to use	Simple baseline	When you have strong prior about weight pattern	When you want automatic decay without tuning many weights

14.8.4 Pitfalls

Watch out:

Weights must sum to 1. If they do not, your forecast is systematically biased — it will overpredict or underpredict on average. Always verify $\sum w_{i} = 1$ .
Too many weights to tune. A 10-period weighted MA has 10 weights to choose. That is 10 degrees of freedom on potentially limited data — you risk overfitting the weight pattern to noise. In practice, weighted MA is most useful with short windows (2–4 periods), and for longer memory you switch to exponential smoothing.
Choosing weights by gut feel. "Recent data is more important, so I will give it 0.5" is a start, but the weights should be validated against holdout error just like the window length $k$ .

14.8.5 Recap and Bridge

Recap: Weighted moving averages let you emphasize recent observations by assigning them larger weights. The weights must sum to 1. This is more flexible than simple MA but introduces more parameters to tune. The next logical step: instead of choosing each weight individually, impose an exponential decay pattern so only one parameter (

α

) controls the entire weight distribution. That is exponential smoothing.

14.8.6 Real-World & Domain Connection

Weighted moving averages are standard in financial technical analysis. The widely-used exponential moving average (EMA) on trading platforms is essentially a weighted MA with exponential decay. Traders compare EMA(12) and EMA(26) — when the faster (12-period) crosses above the slower (26-period), it signals upward momentum. The entire crossover strategy rests on the idea that recent prices should influence the average more than older ones.

14.9 Exponential Smoothing

What if you could let the data decide how much weight to give the past — using a single knob? That is exponential smoothing. One parameter, $α$ , controls the entire weight distribution: recent observations get more weight, older ones get less, and the decay follows a smooth exponential curve. No need to choose individual weights for each lag.

14.9.1 The Core Formula

Imagine you are steering a car and adjusting course based on what you see ahead. If you trust your current view completely ( $α = 1$ ), you jerk the wheel with every new observation. If you trust only your existing direction ( $1 - α = 1$ , so $α = 0$ ), you never turn the wheel at all. Exponential smoothing is driving with a steady hand: you blend the new view with your current trajectory.

The formula:

F_{t + 1} = α Y_{t} + (1 - α) F_{t}

Today's forecast for tomorrow equals a blend of today's actual value and the forecast you had made for today. The parameter $α$ (between 0 and 1) controls the blend.

Where the analogy breaks: a car driver can look far ahead. Exponential smoothing only looks at the immediate next observation through the recurrence — though the recurrence carries information from all past observations with decaying weights.

Exponential Smoothing (Simple):

F_{t + 1} = α Y_{t} + (1 - α) F_{t}

where:

$F_{t + 1}$ = forecast for time $t + 1$ (made at time $t$ )
$Y_{t}$ = actual value observed at time $t$
$F_{t}$ = forecast that was made for time $t$ (computed at $t - 1$ )
$α$ = smoothing parameter, $0 \leq α \leq 1$

The formula is a weighted blend: weight $α$ on the new observation, weight $1 - α$ on the old forecast.

14.9.2 Symbol Registry

Symbol	Meaning	Type	Domain
$Y_{t}$	actual value at time $t$	scalar	$R$
$F_{t}$	forecast for time $t$ (made at $t - 1$ )	scalar	$R$
$F_{t + 1}$	forecast for time $t + 1$	scalar	$R$
$α$	smoothing parameter (weight on actual)	scalar	$[0, 1]$

14.9.3 What $α$ Controls

$α$	Behavior	Use case
0	$F_{t + 1} = F_{t}$ — forecast never changes	Series with no pattern (pure noise); you believe the mean is constant
0.1–0.3	Slow adaptation; smooth forecasts	Stable series with gradual shifts
0.5–0.7	Moderate responsiveness	Balance between stability and reactivity
0.8–0.9	Fast adaptation; tracks changes closely	Volatile series where recent data is highly informative
1	$F_{t + 1} = Y_{t}$ — tomorrow = today (naive forecast)	Random walk; zero smoothing

The choice of $α$ is analogous to the learning rate in gradient descent: too high and you oscillate; too low and you barely move. You tune $α$ by trying different values and picking the one that minimizes forecast error on historical data.

14.9.4 Why It Is Called "Exponential"

Expand the recurrence to see the weight pattern:

F_{t + 1} F_{t} = α Y_{t} + (1 - α) F_{t} = α Y_{t - 1} + (1 - α) F_{t - 1}

Substitute $F_{t}$ into the first equation:

F_{t + 1} = α Y_{t} + (1 - α) [α Y_{t - 1} + (1 - α) F_{t - 1}] = α Y_{t} + α (1 - α) Y_{t - 1} + (1 - α)^{2} F_{t - 1}

Keep substituting all the way back:

F_{t + 1} = α Y_{t} + α (1 - α) Y_{t - 1} + α (1 - α)^{2} Y_{t - 2} + α (1 - α)^{3} Y_{t - 3} + \dots

The weights form a geometric series: $α, α (1 - α), α (1 - α)^{2}, α (1 - α)^{3}, \dots$ . This is the series expansion of an exponential decay function — each weight is $(1 - α)$ times the previous weight. Hence the name exponential smoothing.

The sum of all weights converges to 1 (since $0 < α \leq 1$ ), confirming that the forecast is a proper weighted average of all past observations — with exponentially decaying weights.

14.9.5 Worked Example

Take $α = 0.3$ . Suppose at time $t = 5$ :

Actual value: $Y_{5} = 21$
Previous forecast for time 5: $F_{5} = 20$ (computed at time 4)

Then the forecast for time 6 is:

F_{6} = α Y_{5} + (1 - α) F_{5} = 0.3 \times 21 + 0.7 \times 20 = 6.3 + 14 = 20.3

The new observation (21) pulled the forecast up from 20 to 20.3 — a nudge of exactly $α$ times the gap $(21 - 20 = 1)$ .

Now forecast one more step. Suppose $Y_{6}$ turns out to be 22:

F_{7} = 0.3 \times 22 + 0.7 \times 20.3 = 6.6 + 14.21 = 20.81

The forecast climbs toward the rising actual values, but smoothly — it does not jump all the way to 22.

Sense-check: With $α = 0.3$ , each new observation corrects the forecast by 30% of the error. The remaining 70% is inertia from the past. Over several periods, the forecast converges toward the actual level — faster with larger $α$ , slower with smaller $α$ .

14.9.6 Initialization

To start the recurrence, you need an initial forecast $F_{1}$ . Common choices:

Naive start: $F_{1} = Y_{1}$ — assume the first forecast equals the first observation.
Average start: $F_{1} = \frac{1}{m} \sum_{i = 1}^{m} Y_{i}$ for the first few observations (e.g., $m = 3$ or $m = 5$ ).
Backcasting: fit the model in reverse and use the backcast as the initial value.

For this course, the initial forecast value will be provided or you can assume $F_{1} = Y_{1}$ unless stated otherwise.

14.9.7 Connection to Deep Learning — RMSProp Optimizer

If you have studied RMSProp or Adam in deep learning, you have already used exponential smoothing. The RMSProp optimizer maintains a running average of squared gradients:

E [g^{2}]_{t} = β E [g^{2}]_{t - 1} + (1 - β) g_{t}^{2}

This is structurally identical to exponential smoothing with $α = 1 - β$ . The new squared gradient gets weight $1 - β$ ; the old running average gets weight $β$ . Adam combines two such exponential moving averages — one for the gradient (momentum) and one for the squared gradient (RMSProp).

The same mathematical idea — blend new information with a running estimate — appears in forecasting (exponential smoothing), optimization (RMSProp/Adam), and reinforcement learning (TD-learning). It is a recurring pattern worth recognizing.

14.9.8 Comparison — Simple MA vs. Weighted MA vs. Exponential Smoothing

Dimension	Simple MA	Weighted MA	Exponential Smoothing
Weight pattern	Equal: $1/ k$ each	Analyst-chosen: $w_{1}, w_{2}, \dots$	Exponential decay: $α (1 - α)^{j - 1}$
Parameters	1 ( $k$ )	$k$ (all weights)	1 ( $α$ )
Memory	Drops observations older than $k$	Drops observations older than $k$	Infinite — all past observations contribute (with decay)
Responsiveness	Uniform within window	Skewed toward recent	Smoothly decays
Best for	Quick baseline	Strong prior on weight pattern	Automatic, parsimonious smoothing

14.9.9 Assumptions and Scope

Scope: When exponential smoothing works and when it breaks.

Works for: Series with no trend and no seasonality (simple exponential smoothing). For trend, use Holt's method. For trend + seasonality, use Holt-Winters. These extensions are beyond this course but are logical upgrades.
Breaks when: The series has a strong trend — simple exponential smoothing will systematically lag behind. The series has seasonality — it cannot capture repeating patterns.
$α$ must be tuned: Do not guess. Try a grid of values (0.1, 0.3, 0.5, 0.7, 0.9) and pick the one that minimizes mean absolute error or mean squared error on a validation set.

14.9.10 Pitfalls

Common traps with exponential smoothing:

Setting $α = 1$ and calling it a model. $F_{t + 1} = Y_{t}$ is the naive forecast — it is a special case of exponential smoothing, but it does no actual smoothing.
Forgetting that $α$ near 1 produces jittery forecasts. You are essentially following the last observation. This is only appropriate for random walks. For most business data, $α$ between 0.1 and 0.4 is more realistic.
Confusing $F_{t}$ (forecast for time $t$ ) with $Y_{t}$ (actual at time $t$ ). The formula blends them, so keeping them straight is critical. $F_{t}$ was computed before seeing $Y_{t}$ ; $F_{t + 1}$ is computed after seeing $Y_{t}$ .
Not checking residuals. After fitting, the forecast errors $Y_{t} - F_{t}$ should be random (white noise). If they show a pattern (e.g., all errors are positive — you are underpredicting), the model is misspecified — you likely need trend or seasonality.

14.9.11 Recap and Bridge

Recap: Exponential smoothing produces a forecast as a blend of the latest observation and the previous forecast: $F_{t + 1} = α Y_{t} + (1 - α) F_{t}$ . The parameter $α$ controls responsiveness. Expanding the recurrence reveals that every past observation contributes with exponentially decaying weight. Simple exponential smoothing is the foundation — it handles horizontal patterns. For trend and seasonality, the ARIMA family extends these ideas.

14.9.12 Real-World & Domain Connection

Exponential smoothing is the workhorse of inventory management. Retailers like Walmart use it to forecast demand for hundreds of thousands of SKUs — simple exponential smoothing is computationally trivial and can run automatically across the entire catalog. When a product's sales spike (a new TikTok trend), a tuned $α$ catches the shift within days. When sales are stable, the same model smoothly filters out daily noise. The simplicity is the feature: one parameter per product, no manual weight specification, and forecasts that update in real time as each new sale comes in.

14.9.13 Student Questions and Answers

Q: The exponential smoothing formula $F_{t + 1} = α Y_{t} + (1 - α) F_{t}$ — is it connected to RMSProp in deep learning?

A: Yes, exactly. A student spotted this cross-domain connection. RMSProp uses the same weighted blending: $E [g^{2}]_{t} = β E [g^{2}]_{t - 1} + (1 - β) g_{t}^{2}$ . Structurally, it is exponential smoothing of squared gradients, with $α = 1 - β$ . The Adam optimizer also combines two exponential moving averages. If you have seen RMSProp or Adam, you have already used exponential smoothing — just in a different context (optimization rather than forecasting).

Q: How do you get the initial forecast $F_{1}$ to start the recurrence?

A: The initial forecast is typically set to the first actual observation ( $F_{1} = Y_{1}$ ) or estimated from the first few data points using a simple average. In practice, for this course, the initial forecast value will be provided or you can assume $F_{1} = Y_{1}$ . The choice of initialization matters most for short series — with enough data, the effect of the initial value decays away.

Q: Does exponential smoothing only look one step back, or can it look further?

A: The formula only explicitly references $Y_{t}$ and $F_{t}$ . But $F_{t}$ itself embeds $Y_{t - 1}$ and $F_{t - 1}$ , which embeds $Y_{t - 2}$ and $F_{t - 2}$ , and so on. The model implicitly incorporates all past observations with exponentially decaying weights. It never "forgets" completely — unlike a simple MA, which drops observations older than $k$ . More elaborate versions (double exponential smoothing, Holt's method, Holt-Winters) add explicit trend and seasonality terms that look further back in structured ways.

14.10 Overview of ARIMA, SARIMA, and Related Statistical Models

Moving averages and exponential smoothing are simple and effective — but what if your data has trend, seasonality, and complex autocorrelation all at once? For decades, the answer has been the ARIMA family of statistical models. They are the baseline toolkit for serious time series work, and you need to know what they are even if you do not fit them by hand.

14.10.1 The Family of Time Series Models

The ARIMA family builds from simpler pieces. Here is the progression:

AR (Autoregressive): The value at time $t$ is a linear combination of its own past values plus noise.

Y_{t} = c + ϕ_{1} Y_{t - 1} + ϕ_{2} Y_{t - 2} + \dots + ϕ_{p} Y_{t - p} + ϵ_{t}

Think: "today depends on yesterday, the day before, and so on." Parameters: $p$ (how many lags), $ϕ_{1}, \dots, ϕ_{p}$ (how strongly each lag influences today).

MA (Moving Average): The value at time $t$ depends on past forecast errors, not past values.

Y_{t} = μ + ϵ_{t} + θ_{1} ϵ_{t - 1} + θ_{2} ϵ_{t - 2} + \dots + θ_{q} ϵ_{t - q}

Think: "today is the mean plus today's shock, adjusted by how much past shocks still echo." Parameters: $q$ (how many past error terms), $θ_{1}, \dots, θ_{q}$ .

Important: this MA ("Moving Average" of errors) is NOT the same as the simple moving average of values from section 14.7. Same name, completely different concept.

ARMA: Combines AR and MA. $Y_{t}$ depends on both its own past values and past errors.

Y_{t} = c + AR (p) ϕ_{1} Y_{t - 1} + \dots + ϕ_{p} Y_{t - p} + MA (q) ϵ_{t} + θ_{1} ϵ_{t - 1} + \dots + θ_{q} ϵ_{t - q}

ARIMA (Autoregressive Integrated Moving Average): ARMA applied to differenced data. The "I" stands for "Integrated" — the order of differencing, $d$ .

ARIMA (p, d, q) : Δ^{d} Y_{t} follows an ARMA (p, q) model

where $Δ Y_{t} = Y_{t} - Y_{t - 1}$ (first difference). Differencing removes trend: if $Y_{t}$ grows linearly, $Δ Y_{t}$ is roughly constant. ARIMA makes the series stationary before modeling.

SARIMA (Seasonal ARIMA): ARIMA with seasonal terms added at the seasonal lag $s$ .

SARIMA (p, d, q) (P, D, Q)_{s}

The uppercase $(P, D, Q)$ are the seasonal AR, differencing, and MA orders, applied at lags $s, 2 s, \dots$ (e.g., lag 12 for monthly data with yearly seasonality).

SARIMAX: SARIMA with eXogenous variables — external predictors beyond the time series itself (e.g., include advertising spend when forecasting sales).

GARCH (Generalized Autoregressive Conditional Heteroskedasticity): Models volatility — the variance of the error term changes over time. Used heavily in finance where calm periods alternate with turbulent periods (volatility clustering). GARCH does not forecast the level; it forecasts the risk.

14.10.2 Why These Models Matter

ARIMA models have been in use since the 1970s (Box-Jenkins methodology). They remain relevant because:

Interpretable: each parameter has a clear meaning — $ϕ_{1}$ tells you how strongly yesterday influences today.
Small-data friendly: they work on datasets with as few as 30–50 observations, where deep learning would overfit.
Computationally cheap: fitting an ARIMA takes seconds on a laptop.
Still the default in business: many ERP and supply chain systems have ARIMA built in.

Their limitations:

Assume linear relationships. Cannot capture nonlinear dynamics.
Struggle with long-range dependencies (values separated by hundreds of time steps).
Require manual specification of $(p, d, q)$ — model order selection is part art, part science (ACF/PACF plots).
Univariate by default — extensions like VAR are needed for multiple series.

These limitations drive the move to deep learning (LSTMs, Transformers) for complex forecasting. But ARIMA is still the baseline you compare against.

14.10.3 Industry Perspective — Predictive Maintenance

The professor shared a real consulting story. A manufacturer supplying components to Boeing was losing roughly 250–300 crore rupees annually due to sudden machine failures and emergency servicing. They had sensor data — minute-by-minute readings from every machine in the production line. The goal: predict degradation before failure so maintenance could be planned instead of rushed.

The company had used ARIMA-based models since the 1970s. Senior leadership — IIT graduates who had implemented those models themselves — were skeptical: "We already did ARIMA at IIT Kanpur in the 1970s. What more can you offer?"

The answer: machine learning models could capture nonlinear degradation patterns that ARIMA's linear assumptions missed. They could handle high-dimensional sensor data (hundreds of sensors per machine). They could adapt to changing machine behavior without manual re-specification of model order. The conversation took nearly an hour — not because the technical argument was complex, but because the existing solution had 40 years of trust behind it.

The takeaway: statistical models are not obsolete. They are the baseline. ML and DL models build on them, not replace them. In any forecasting project, you fit an ARIMA first. If it is good enough, stop. If not, you have a benchmark to beat.

14.10.4 Comparison — Statistical vs. ML/DL Time Series Models

Dimension	Statistical (ARIMA family)	ML/DL (LSTM, Transformer)
Data requirement	30–50+ observations	Thousands to millions
Interpretability	High — every parameter has meaning	Low — black box
Nonlinear patterns	Cannot capture	Can capture
Multivariate	Extensions exist (VAR, SARIMAX)	Native support
Training time	Seconds	Minutes to hours (GPU)
Model selection	Manual (ACF/PACF, information criteria)	Automated (hyperparameter tuning)
Best for	Small data, interpretable forecasting, baseline	Large data, complex patterns, high accuracy

14.10.5 Visual Intuition

An AR(1) model $Y_{t} = 0.8 Y_{t - 1} + ϵ_{t}$ produces a series that meanders — each value is 80% of the previous value plus random noise. The series has "memory": a shock today lingers for several periods. An MA(1) model $Y_{t} = ϵ_{t} + 0.8 ϵ_{t - 1}$ has shorter memory — a shock today only affects tomorrow, then vanishes. Plot them side by side: the AR(1) has slow, smooth swings; the MA(1) has sharp, isolated blips.

The takeaway: AR terms create persistence; MA terms create short-lived echoes of past shocks. ARIMA combines both after removing trend through differencing.

14.10.6 Pitfalls

What to watch for (at the conceptual level):

Confusing MA (Moving Average of errors) with the simple moving average from section 14.7. Same words, completely different math. The MA in ARIMA is about past errors, not past values.
Over-differencing. One difference ( $d = 1$ ) removes a linear trend. Two differences ( $d = 2$ ) remove a quadratic trend. More differencing than needed introduces artificial patterns. Most real series need $d = 0$ or $d = 1$ .
Assuming ARIMA is always better than simple methods. For short-horizon forecasts of noisy series, simple exponential smoothing often matches or beats ARIMA. Do not overcomplicate without evidence.
Ignoring the "I" — not checking stationarity. ARMA assumes the series is stationary (constant mean and variance over time). If it is not, you must difference first — that is what the "I" in ARIMA does.

14.10.7 Recap and Bridge

Recap: The ARIMA family — AR (past values), MA (past errors), I (differencing) — is the statistical backbone of time series forecasting. SARIMA adds seasonality; GARCH models volatility. These models are interpretable, data-efficient, and still the industry baseline. For this course: know the names, the basic logic, and the limitations. No deep numericals on ARIMA fitting.

Exam note: ARIMA, SARIMA, and GARCH appear at the conceptual level only. You do not need to fit these models. Know what each acronym stands for, what problem each solves, and the rough progression AR → MA → ARMA → ARIMA → SARIMA.

14.10.8 Real-World & Domain Connection

ARIMA is embedded in the forecasting engines of SAP, Oracle, and most ERP systems. When a supply chain manager clicks "forecast demand" for the next quarter, an ARIMA variant is likely running underneath. Central banks use SARIMA to forecast inflation and GDP — the seasonal adjustment of economic data is essentially a SARIMA decomposition. In energy, grid operators forecast hourly electricity demand with SARIMA models that capture both the daily cycle (people use more electricity at 6 PM than 3 AM) and the yearly cycle (more electricity in summer for AC, in winter for heating). These are essential forecasts where interpretability matters — if the forecast is wrong, you need to explain why to regulators.

Exam Guidance Summary

Exam note: The final exam is roughly 80% post-mid-semester material (hypothesis testing, time series) and 20% pre-mid-semester material (correlation and regression). A consolidated formula sheet and problem set will be shared before the exam. A dedicated revision session will cover problem-solving strategies.

Regression Module (20% weight)

Problem type	What to do
Covariance and correlation	Compute both from given data. Compare what covariance tells you (direction, units) vs. correlation (direction, strength, normalized). Comment on the relationship.
Simple linear regression	Fit $Y = w_{0} + w_{1} X$ . Predict $Y$ for a given $X$ . Comment on the trend — directly proportional, inversely proportional, linear?
$R^{2}$ interpretation	$R^{2} = 0.2$ → poor fit, data likely nonlinear. $R^{2} = 0.8$ → good fit, data predominantly linear. Explain what causes low $R^{2}$ .
Linear vs. nonlinear decision	Examine scatterplot, try linear fit, check $R^{2}$ , assess residuals for systematic curvature. Explain your strategy.

Time Series Module (80% weight)

Problem type	What to do
Moving averages	Compute 3-year and 5-year centered moving averages from tabulated data.
Model selection	Compare MA(3) vs. MA(5) using absolute error $∣ Y_{t} - Y^_{t} ∣$ . Identify which is better and justify with numbers.
Weighted moving average	May appear. Apply weights that sum to 1. Multiply each value by its weight before summing.
Exponential smoothing	Apply $F_{t + 1} = α Y_{t} + (1 - α) F_{t}$ . Understand what different $α$ values mean ( $α \to 0$ : smooth, unresponsive; $α \to 1$ : tracks observations directly).
Time series components	Identify and interpret trend, seasonality, cyclicality, and irregularity in a described or plotted series.

Exam note: ARIMA, SARIMA, and GARCH appear at the conceptual level only. Know the names, what each acronym stands for, and the basic progression (AR → MA → ARMA → ARIMA → SARIMA). No model fitting, no numericals on these topics.

General Exam Tips

Inference questions ("Comment on the trend," "Which model is better and why?") require justification, not just answers. Reference your computed numbers.
Assumption-based questions: write out all assumptions explicitly. Show every computational step — partial credit depends on visible reasoning.
The formula sheet will be provided. Focus on understanding when and why to apply each formula, not on memorizing them.

Key Industry Applications

The concepts in this lecture are not academic exercises — they are deployed in production across industries every day.

Manufacturing

Predictive maintenance uses sensor data from production lines to forecast machine failures before they happen. One Boeing component supplier was losing 250–300 crore rupees annually to unplanned downtime. Their ARIMA-based statistical models (in place since the 1970s) were the baseline; the shift to ML models was driven by the need to capture nonlinear degradation patterns. The core idea: time series forecasting of vibration, temperature, and pressure readings → early warning of impending failure → planned maintenance instead of emergency repairs.

Finance and Trading

Moving averages are the most visible time series tool in the world. Every trading platform overlays MA(3), MA(5), MA(50), and MA(200) on price charts. Traders watch for crossovers — when a short MA crosses above a long MA, it signals upward momentum. Exponential smoothing (called EMA — Exponential Moving Average — in finance) gives more weight to recent prices. GARCH models volatility for options pricing and risk management. The gold and silver markets show long-term upward trends punctuated by sharp drops — exponential smoothing with a tuned $α$ balances responsiveness against trend-following.

Retail and Supply Chain

Seasonality drives inventory decisions. Refrigerator sales spike every summer. Gold purchases rise near Diwali. US retail jumps every December. Retailers use seasonal decomposition to separate the holiday bump from genuine growth, then stock accordingly. Exponential smoothing runs behind the scenes in demand forecasting systems that manage hundreds of thousands of SKUs.

Government and Policy

Population projections over decades use trend analysis. India's Planning Commission and NITI Aayog look at century-scale demographic trends to plan infrastructure — schools, hospitals, roads — deliberately ignoring month-to-month noise. Pollution monitoring agencies decompose air quality data to separate long-term improvement from irregular events like COVID lockdowns, which produced near-zero readings that would distort trend estimates if included.

Agriculture

Crop yields follow seasonal cycles — wheat in April, rice later in the year — with weather-driven irregularity that must be modeled, not removed. Unlike a one-off pollution anomaly, agricultural irregularity is structural: every season brings unpredictable weather, and the forecast must account for that uncertainty.

Health and Wearables

BP readings, step counts, heart rate — all arrive as time series from wearable devices. The first modeling decision is always: does yesterday's reading influence today's? If yes (as with most physiological data), time series methods apply. If not (as with independent diagnostic measurements), regression may suffice.

ISM Lecture 14 notes · Time Series Analysis — Foundations and Basic Forecasting Models

Introduction to Statistical Methods· postgraduate· 2026-07-07

Sections Breakdown

1Multicollinearity

Definition, detection using VIF, and remedies for highly correlated predictor variables in regression models.

2Overfitting and Regularization

Understanding overfitting and using L1 (Lasso), L2 (Ridge), and Elastic Net regularization to control model complexity.

3Exam Guidance — Correlation and Regression Recap

Exam strategy for the correlation and regression module, including weight distribution and expected question types.

4Time Series vs. Regression Models

The core distinction between static regression models and dynamic time series models, including the random split problem.

5Components of a Time Series

Decomposing a time series into trend, seasonality, cyclicality, and irregularity components.

6Additive and Multiplicative Models

Two ways to combine time series components: additive (sum) and multiplicative (product) models.

7Simple Moving Average (MA) Models

Smoothing time series data using centered and trailing moving averages with various window lengths.

8Weighted Moving Average

Assigning higher weights to recent observations for more responsive forecasts.

9Exponential Smoothing

One-parameter smoothing using exponentially decaying weights, with connections to RMSProp in deep learning.

10Overview of ARIMA, SARIMA, and Related Statistical Models

The ARIMA family of statistical time series models: AR, MA, ARMA, ARIMA, SARIMA, and GARCH.

Postgraduate students in Introduction to Statistical Methods

Exam Revision Notes

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review — but if something doesn't make sense, go back to the full explanation in the main content.

Multicollinearity

Must-know: Multicollinearity occurs when predictor variables are highly correlated, inflating coefficient variance. Detect it using VIF: if VIF > 5, investigate and drop or combine variables. The fix is always the same — remove one of the correlated predictors.

$VIF_{j} = \frac{1}{1 - R _{j}^{2}}$

⚠️ Top pitfall: Dropping the causally important variable instead of the redundant one. VIF tells you there is a problem — domain knowledge tells you which variable to keep.

Self-check: Why does a high VIF make hypothesis tests and confidence intervals unreliable?

Connects to: Overfitting and Regularization, Linear Regression

Overfitting and Regularization

Must-know: Overfitting is memorizing noise, not learning patterns. Regularization — L1 (Lasso), L2 (Ridge), or Elastic Net — penalises large weights to improve generalization. Lasso zeros out irrelevant features (sparsity); Ridge shrinks all features; Elastic Net blends both. Always standardize features before regularizing.

$w min i = 1 \sum m (y_{i} - y^_{i})^{2} + λ_{1} j = 0 \sum n ∣ w_{j} ∣ + λ_{2} j = 0 \sum n w_{j}^{2}$

⚠️ Top pitfall: Not standardizing features before regularizing — features with larger scales get penalised disproportionately.

Self-check: Why does the L1 norm (diamond-shaped constraint region) produce sparse solutions while L2 (circle-shaped) does not?

Connects to: Multicollinearity, Lagrange Multipliers, Linear Regression

Time Series vs. Regression Models

Must-know: Regression assumes i.i.d. data — shuffle freely. Time series data has order-dependent structure — shuffling destroys the signal. Use chronological (not random) train/test splits for time series. The diagnostic question: does yesterday’s value influence today’s?

$Y = f (X) (regression, static) vs. Y_{t} depends on Y_{t - 1}, Y_{t - 2}, \dots (time series, dynamic)$

⚠️ Top pitfall: Assuming any data with a timestamp is a time series. Validate whether Y_t genuinely depends on t — a patient measurement dataset with dates is cross-sectional, not time series.

Self-check: You have 12 months of sales data and want to forecast month 13. How should you split the data for training and testing?

Connects to: Components of a Time Series, Moving Average Models, ARIMA

Components of a Time Series

Must-know: Every time series decomposes into trend (long-run direction), seasonality (fixed-period cycles — analyst-defined), cyclicality (variable-period waves — observed from data), and irregularity (random noise). If you can put the period on a calendar, it is seasonal; otherwise, it is cyclical.

$Y_{t} = trend T_{t} + seasonal S_{t} + cyclical C_{t} + irregular I_{t}$

⚠️ Top pitfall: Confusing seasonality (fixed known period, e.g., every December) with cyclicality (variable period, e.g., business cycles every 7–10 years). Students often call everything “seasonal.”

Self-check: Refrigerator sales spike every summer. Is that seasonality or cyclicality? What about economic recessions every 7–10 years?

Connects to: Additive and Multiplicative Models, Time Series vs. Regression, Moving Average Models

Additive and Multiplicative Models

Must-know: Additive model: Y = T + S + C + I — seasonal amplitude is constant over time. Multiplicative model: Y = T × S × C × I — seasonal amplitude scales with trend. Use multiplicative when the plot shows a megaphone (fanning) shape. Log transform converts multiplicative to additive: log(Y) = log(T) + log(S) + log(C) + log(I).

$lo g (Y_{t}) = lo g (T_{t}) + lo g (S_{t}) + lo g (C_{t}) + lo g (I_{t})$

⚠️ Top pitfall: Using additive model when the data fans out — you will underpredict peaks at later times. Taking log of zero or negative values (log(0) is undefined).

Self-check: Your time series plot shows a megaphone shape — peaks get taller as the series rises. Which model should you use and why?

Connects to: Components of a Time Series, Moving Average Models

Simple Moving Average (MA) Models

Must-know: A moving average smooths data by averaging a sliding window of k consecutive values. Centered MA is for model-fitting (uses past and future data); trailing MA is for forecasting (uses only the most recent k known values). Choose k by comparing absolute error |Y_t − Ŷ_t| across candidates — not by eyeballing the graph.

$Y^_{t} = \frac{Y _{t - 1} + Y _{t} + Y _{t + 1}}{3} (3-year centered MA)$

⚠️ Top pitfall: Using centered MA for forecasting — it uses future data (Y_{t+1}) that does not exist when making real predictions. Always switch to trailing MA for out-of-sample forecasts.

Self-check: For a 5-year centered MA on 10 years of data, which years receive no forecast?

Connects to: Weighted Moving Average, Exponential Smoothing, Components of a Time Series

Weighted Moving Average

Must-know: Assigns higher weights to recent observations and lower weights to older ones. Weights must sum to 1 (∑w_i = 1). More flexible than simple MA but introduces more parameters to tune. Best used with short windows (2–4 periods); for longer memory, switch to exponential smoothing.

$Y^_{t + 1} = w_{1} Y_{t} + w_{2} Y_{t - 1} + w_{3} Y_{t - 2}, i = 1 \sum 3 w_{i} = 1$

⚠️ Top pitfall: Weights not summing to 1 — causes systematic bias in forecasts. Too many weights to tune with limited data risks overfitting the weight pattern to noise.

Self-check: Why might a weighted 3-day MA (w₁=0.7, w₂=0.2, w₃=0.1) produce a higher forecast than an equal-weight 3-day MA when prices are trending upward?

Connects to: Simple Moving Average, Exponential Smoothing

Exponential Smoothing

Must-know: F_{t+1} = αY_t + (1−α)F_t — blend the new observation with the previous forecast. α controls responsiveness: α → 0 = very smooth (unresponsive), α → 1 = tracks observations directly (naive forecast). Expanding the recurrence reveals exponentially decaying weights on all past observations. Same mathematical structure as RMSProp in deep learning.

$F_{t + 1} = α Y_{t} + (1 - α) F_{t}$

⚠️ Top pitfall: Confusing F_t (forecast for time t, made at t−1) with Y_t (actual at time t). Setting α = 1 (naive forecast) and calling it a model — that does no actual smoothing.

Self-check: With α = 0.3, Y_t = 21, and F_t = 20, compute F_{t+1}. What does the result tell you about how α controls the forecast correction?

Connects to: Simple Moving Average, Weighted Moving Average, ARIMA

ARIMA, SARIMA, and Related Models

Must-know: The progression: AR (past values) → MA (past errors) → ARMA (both) → ARIMA (adds differencing I for stationarity) → SARIMA (adds seasonality). GARCH models volatility. For this course: know the names, what each acronym stands for, and the basic logic. No model fitting required.

$ARIMA (p, d, q) : Δ^{d} Y_{t} follows ARMA (p, q), Δ Y_{t} = Y_{t} - Y_{t - 1}$

⚠️ Top pitfall: Confusing the MA in ARIMA (moving average of past errors) with the simple moving average of values from Section 14.7. Same name, completely different concept.

Self-check: What does the “I” in ARIMA stand for, and what problem does it solve?

Connects to: Exponential Smoothing, Moving Average Models, Components of a Time Series

Previous LectureLecture 13 · Covariance, Correlation, and Linear Regression

Next LectureLecture 15 · Time Series Analysis — Smoothing and Forecasting

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.