Time Series Analysis — Smoothing and Forecasting

What connects every method in this lecture is one idea: the future resembles the past, but only the recent past matters most. Moving averages, exponential smoothing, weighted variants, Holt's trend-tracking method — each is a different bet on how much of the past to carry forward and how fast to let it fade.

Time series forecasting sits at the intersection of statistics and practical decision-making. Businesses use it for inventory planning. Economists use it for GDP projections. Engineers use it for signal processing. The methods here are the foundation that every more advanced technique — ARIMA, state-space models, deep learning forecasters — builds on.

The lecture closes with an introduction to Gaussian Mixture Models, a probabilistic clustering framework. It may seem like a detour from forecasting, but the thread is the same: extracting structure from data, whether that structure is a trend over time or hidden groups in a cloud of points.

15.1 Moving Averages — Review and Extension

15.1.1 Definition and Core Idea

A moving average replaces the current value with the average of a fixed-size window of past observations. Slide a window across the time series. For each window position, compute the average. That average becomes the forecast. The window size $k$ is your key design choice — it controls how many time steps you include. You use moving averages to smooth short-term fluctuations and reveal longer-term trends.

15.1.2 Symbol Registry

15.1.3 Odd vs Even Window Sizes — The Key Difference

15.1.4 Assumptions & Scope

15.1.5 Visual Intuition

Plot your raw time series as connected dots $(t,Y_t)$. Now overlay the moving average as a second line. The MA line will be smoother — the peaks are lower, the troughs are higher. It cuts through the middle of the fluctuations. For a 3-period MA, count: the MA line at year 2 is the average of years 1, 2, and 3. The MA line at year 3 averages years 2, 3, and 4. The line trails the raw data by roughly half a window width. The wider the window (larger $k$), the flatter and laggier the MA line becomes.

15.1.6 Pitfalls

15.1.7 Student Questions and Answers

15.1.8 Recap + Bridge

15.1.9 Real-World & Domain Connection

Moving averages are the workhorse of technical analysis in finance. Traders watch the "50-day MA" and "200-day MA" cross each other as buy/sell signals — when the short-term MA crosses above the long-term MA, it's a "golden cross" (bullish). Beyond finance, moving averages appear in signal processing (as low-pass filters), in epidemiology (7-day rolling averages of case counts became a household concept during COVID-19), and in any domain needing a quick, interpretable baseline forecast. They survive because they require no distributional assumptions, no optimization, and can be explained to a non-technical stakeholder in thirty seconds.

15.2 Exponential Smoothing

15.2.1 Definition and Core Idea

15.2.2 The Formula

15.2.3 Symbol Registry

15.2.4 Why "Exponential" — The Infinite Expansion

15.2.5 Handling the First Forecast

At the very first time step, you have $Y_1$ but no $F_1$. The convention: set $F_1=Y_1$. With no prior information, you start with the observed value as the initial forecast. This matches the textbook approach: the forecast for period 2 is $F_2=\alpha Y_1+(1-\alpha )F_1=\alpha Y_1+(1-\alpha )Y_1=Y_1$, so effectively the first real forecast is just the first observation. The error-correction view reinforces this — without a prior forecast, there is no error to correct.

15.2.6 Worked Example 1 — Exponential Smoothing with $\alpha =0.2$

15.2.7 Worked Example 2 — Comparing $\alpha =0.2$, $\alpha =0.4$, $\alpha =0.7$

15.2.8 Assumptions & Scope

15.2.9 Visual Intuition

Plot your raw time series $(t,Y_t)$ as a jagged line. Overlay the exponential smoothing forecast $(t,F_t)$. At $\alpha =0.2$, the forecast line is a heavily damped version — it drifts slowly, ignoring most of the wiggles. At $\alpha =0.7$, the forecast line hugs the actuals tightly, jumping with every spike. Think of $\alpha$ as a "responsiveness knob." Turn it to 0: the line goes flat. Turn it to 1: the line becomes the raw data shifted right by one period.

15.2.10 Pitfalls

15.2.11 Student Questions and Answers

15.2.12 Interpreting Alpha — Small vs Large Values

Small alpha (e.g., 0.1–0.3): The forecast is stable and slow to change. It gives more weight to accumulated history than to the latest observation. Good when the underlying process is stable and noise is high. The forecast line will be relatively flat.

Large alpha (e.g., 0.7–0.9): The forecast is reactive — it tracks the latest observation closely. Good when the process changes rapidly and you need to respond fast. The forecast line hugs the actual data.

15.2.13 Recap + Bridge

15.2.14 Real-World & Domain Connection

Exponential smoothing powers production forecasting systems worldwide. Amazon's inventory planning, Walmart's supply chain, and airline revenue management all use variants of it — often not because it's the most accurate method, but because it is fast, interpretable, and robust to missing data. It requires storing only two numbers (the last actual and the last forecast), making it ideal for high-frequency streaming data. The error-correction form connects directly to modern optimization: the RMSprop optimizer in deep learning uses a similar idea — an exponentially weighted moving average of squared gradients — to adapt learning rates. The same mathematical idea, separated by six decades of application.

15.3 Performance Measures for Forecast Evaluation

15.3.1 Why You Need Performance Measures

You cannot judge forecast models by intuition. You need a single number summarizing how wrong predictions are across all time steps. Different metrics emphasize different aspects of error.

15.3.2 Symbol Registry

15.3.3 Mean Squared Error (MSE)

15.3.4 Mean Absolute Deviation (MAD)

15.3.5 Mean Absolute Percentage Error (MAPE)

15.3.6 Largest Absolute Deviation (LAD)

15.3.7 Comparison of Metrics

15.3.8 Worked Example — Six Models Compared

15.3.9 Student Questions and Answers

15.3.10 Exam Notes

15.3.11 Recap + Bridge

15.3.12 Real-World & Domain Connection

Every predictive modeling project — from weather forecasting to stock trading to demand planning — requires choosing between candidate models using quantitative criteria. The four metrics here are universal. In Kaggle competitions, MSE (or its square root, RMSE) is the default loss function. In supply chain management, MAPE dominates because it allows comparing forecast accuracy across products with vastly different sales volumes. In aerospace engineering, LAD matters because a single large trajectory error is unacceptable regardless of average performance. Knowing which metric to optimize is half the job of forecasting.

15.4 Weighted Moving Averages

15.4.1 Definition

A weighted moving average is like a regular moving average, but observations inside the window are not treated equally. You assign a weight $w_i$ to each position. The most recent observation typically gets the highest weight. The weights must sum to 1.

15.4.2 Symbol Registry

Constraint: ${\sum }_{i=1}^k{w}_i=1$.

15.4.3 Formula and Computation

15.4.4 Key Rules

• Weights must be between 0 and 1.
• Weights must sum to exactly 1.
• Weight assignment is a design choice — it comes from domain knowledge or experimentation, not from a formula.
• The rest of the procedure (sliding the window, error analysis) is identical to the unweighted moving average.

15.4.5 Assumptions & Scope

15.4.6 Pitfalls

15.4.7 Student Questions and Answers

15.4.8 Recap + Bridge

15.4.9 Real-World & Domain Connection

Weighted moving averages are standard in technical stock analysis. The "exponential moving average" (EMA) used by traders is essentially exponential smoothing — it's a weighted moving average over all past data with geometrically decaying weights. Short-term traders use 12-day and 26-day EMAs; their crossover generates buy/sell signals. In supply chains, weighted moving averages help forecast perishable goods where last week's demand is objectively more informative than demand from a month ago. In weather forecasting, weighted averages of recent observations feed into nowcasting systems that predict the next hour's conditions.

15.5 Double Exponential Smoothing (Holt's Method)

15.5.1 Motivation — Beyond Single Smoothing

Single exponential smoothing only handles the level of the series. When a time series has a trend — a persistent upward or downward movement — single smoothing systematically lags. Holt's double exponential smoothing adds a second smoothed component for the trend.

15.5.2 The Two Equations

15.5.3 Symbol Registry

15.5.4 Connection to Single Exponential Smoothing

If you remove the trend term $T_{t-1}$ from the level equation:

$$<p>L_t=\alpha Y_t+(1-\alpha )L_{t-1}</p><p>$$

This is exactly single exponential smoothing, with $L_t$ playing the role of $F_{t+1}$. Holt's method is single exponential smoothing plus trend tracking.

15.5.5 Initialization

For the first time step:

• Set $L_1=Y_1$ — same convention as single smoothing: no prior level available.
• Set $T_1=0$ — with only one data point, you cannot estimate a trend.

Notation note: The textbook (Anderson et al.) uses a different initialization: $L_1=Y_1$ and $b_1=Y_2-Y_1$ (the first observed difference as the initial trend). The professor's convention ($T_1=0$) is simpler and the one the exam expects. Both are valid; they converge after a few periods. Use $T_1=0$ for exam computations.

15.5.6 Worked Numerical Example

15.5.7 Extreme Values of Alpha and Beta

15.5.8 Assumptions & Scope

15.5.9 Visual Intuition

Plot the raw time series $(t,Y_t)$ as dots. Overlay three lines:

1. The level $L_t$ — a smoothed version of the raw data, similar to single exponential smoothing.
2. The trend $T_t$ — not directly visible on the same scale; plot it on a secondary axis or note its growth in a table.
3. The forecast $F_{t+1}=L_t+T_t$ — this line leads the raw data. In an uptrend, it sits above the most recent actual, anticipating the next rise. The gap between $F_{t+1}$ and $Y_t$ is the one-step-ahead trend correction.

For the GDP example: $F_{2002}=1504.76$ while $Y_{2001}=1499$. The forecast is higher — Holt's "knows" GDP is trending up and builds that expectation into the prediction.

15.5.10 Pitfalls

15.5.11 Student Questions and Answers

15.5.12 Time Series Decomposition — The Pre-Processing Step

Before choosing among single, double, or triple smoothing, decompose your time series into components: trend, seasonality, and residual (irregular noise). Plot the data. Look for an upward or downward drift (trend). Look for regular repeating waves (seasonality). If decomposition shows a clear trend, use double smoothing. If it also shows seasonality, use triple smoothing. If neither, single smoothing may be enough.

15.5.13 Broader Context — Triple Exponential Smoothing

If your time series has a seasonal component, you need triple exponential smoothing (Holt-Winters). It adds a third equation for seasonality alongside level and trend. Triple smoothing is not on the exam, but the key takeaway is: the exponential smoothing framework scales. Add components as your data's complexity demands.

15.5.14 Recap + Bridge

15.5.15 Real-World & Domain Connection

Holt's method (and its seasonal extension, Holt-Winters) powers operational forecasting across industry. Retailers use it for weekly sales forecasts that capture both the upward trend of a growing business and the seasonal spikes of holiday shopping. Energy companies forecast electricity demand with it — the trend captures economic growth, and an added seasonal component captures daily and weekly cycles. Airlines forecast passenger volumes. The method is embedded in SAP, Oracle, and most ERP systems' forecasting modules. Its durability comes from a rare combination: it handles the two most common time series features (level changes and trends) with only two parameters, runs in constant memory, and produces forecasts that a business analyst can explain to a VP.

15.6 Gaussian Mixture Models — Introduction

15.6.1 Motivation — From Unlabeled Data to Patterns

A Gaussian Mixture Model (GMM) is a probabilistic model for clustering. It assumes your data came from a mixture of several Gaussian distributions. Each Gaussian has its own mean ${\mu }_k$ and covariance ${\Sigma }_k$. The goal: recover those underlying Gaussians. That tells you which cluster each data point likely came from.

Why Gaussians? The Central Limit Theorem pushes most aggregated phenomena toward normality. The Gaussian is the most mathematically tractable distribution and the natural default.

15.6.2 The Clustering Analogy — Hard vs. Soft

15.6.3 From Univariate to Multivariate Gaussians

15.6.4 The Covariance Matrix — Why It Matters

When you have a single variable, you have variance ${\sigma }^2$ — the spread. With multiple variables, you also have covariance — how two variables move together.

For $d$ variables, $\Sigma$ is $d\times d$:

• Diagonal: ${\sigma }_1^2,{\sigma }_2^2,\dots ,{\sigma }_d^2$ — variances of each variable.
• Off-diagonal: $\text{Cov}(X_i,X_j)$ — covariance between variables $i$ and $j$.

$\Sigma$ is a matrix, not the summation symbol $\sum$. It captures all pairwise relationships. Its determinant $|\Sigma |$ measures the generalized "volume" of the distribution. Its inverse ${\Sigma }^{-1}$ appears in the Mahalanobis distance, rotating and scaling space so that the distribution becomes a standard spherical Gaussian.

15.6.5 The Overlap Problem — Why GMM Gets Interesting

Consider two Gaussian clusters with different means and overlapping tails. A data point falls in the overlap region. Which cluster does it belong to? Was it drawn from distribution A or B? There is genuine ambiguity. GMM addresses this by estimating responsibilities — the probability that a given point came from each component. Points in clear regions get near-certain assignments ($p\approx 1$). Points in the overlap get fractional probabilities ($p\approx 0.5$).

15.6.6 A GMM Formally

15.6.7 Assumptions & Scope

15.6.8 Visual Intuition

Picture a 2D scatter plot. k-means draws Voronoi cells — straight-line boundaries partitioning the plane into exclusive regions. GMM draws nested ellipses — each ellipse is a contour of constant probability for one Gaussian component. Where ellipses overlap, both components claim the points inside. The ellipses can be circles (spherical covariance), axis-aligned ellipses (diagonal covariance), or tilted ellipses (full covariance) — the shape depends on ${\Sigma }_k$.

For the hill-station analogy: draw three overlapping bell curves along a 1D axis. Each curve peaks at its mean. At any $x$, the height of the mixture is the sum of the three curves' heights, weighted by ${\pi }_k$. The dominance at $x$ is determined by which curve is tallest there.

15.6.9 Pitfalls

15.6.10 Recap + Bridge

15.6.11 Real-World & Domain Connection

GMMs power customer segmentation in marketing, grouping shoppers by purchase behavior. They detect anomalies in fraud systems, where points with low probability under the mixture are flagged. In computer vision, they segment images by modeling pixel colors as a mixture of Gaussians. In speech recognition, phoneme distributions were modeled as GMMs — the dominant approach before deep learning. In bioinformatics, GMMs cluster gene expression data to discover functionally related groups of genes. The soft-clustering property is especially valuable in medicine. A patient's symptoms may reflect multiple underlying conditions at once, something hard clustering cannot model.

Exam Guidance Summary

The exam covers:

• Moving averages — both odd and even window sizes. Know the placement rule: odd $k$ → middle; even $k$ → $k$-th position. Expect to compute 2-quarter, 3-quarter, 4-quarter, 2-year, 3-year, 4-year, and 5-year moving averages.
• Weighted moving averages — weights must sum to 1 and be between 0 and 1. Expect computation with given weights.
• Single exponential smoothing — know both forms: $F_{t+1}=\alpha Y_t+(1-\alpha )F_t$ and the error-correction form $F_{t+1}=F_t+\alpha (Y_t-F_t)$. Expect numerical problems where you forecast with different alpha values and interpret which is better. Know why $F_1=Y_1$.
• Double exponential smoothing (Holt's method) — know the level equation $L_t=\alpha Y_t+(1-\alpha )(L_{t-1}+T_{t-1})$ and the trend equation $T_t=\beta (L_t-L_{t-1})+(1-\beta )T_{t-1}$. Expect a numerical where you compute $L_t$, $T_t$, and $F_{t+k}$ with given $\alpha$ and $\beta$. Know why $T_1=0$.
• Performance measures — MSE, MAD, MAPE, LAD. Know how to compute each and what they emphasize. Expect interpretation: given errors, which model is best?
• Interpretation questions — what does a small alpha mean? Large alpha? Why is the first trend zero? Why is the first forecast equal to the first actual? What does $\beta =0$ do to Holt's method?

Not on the exam: Triple exponential smoothing (Holt-Winters); deep statistical theory of centering for moving averages; the EM algorithm for GMM fitting; detailed mathematical derivation of the multivariate Gaussian.

Format: Numerical problems dominate. Show all work in tables. Write assumptions in full detail.

Study advice: Focus on the worked examples. Redo the computations yourself. The six-model alpha comparison (with MSE calculation for each) and the Holt's GDP example are excellent practice problems. For each method, be able to: (1) state the formula, (2) initialize correctly, (3) compute step-by-step in a table, (4) evaluate with MSE/MAD, and (5) justify which model is best.

Key Industry Applications

• Moving averages: Financial analysis (stock price smoothing, technical indicators like golden cross / death cross), demand forecasting, economic trend analysis, epidemiology (7-day rolling averages of case counts), and any domain needing a simple, interpretable baseline forecast. Moving averages are also fundamental building blocks in signal processing as low-pass filters.

• Exponential smoothing: Inventory management, supply chain forecasting, and any streaming data context where you need a lightweight model that adapts to recent changes. Used in production forecasting systems (SAP, Oracle ERP) where simplicity and interpretability matter more than sophistication. The error-correction form connects directly to modern optimization methods like RMSprop in deep learning.

• Weighted moving averages: Situations where recent observations are objectively more relevant — perishable goods demand forecasting, short-term weather nowcasting, social media trend tracking, and financial technical analysis (the "exponential moving average" or EMA used by traders).

• Double exponential smoothing (Holt's method): Forecasting series with clear trends — GDP growth projections, population forecasting, technology adoption curves, retail sales with year-over-year growth, and energy demand forecasting. Holt-Winters (triple exponential smoothing) extends this to seasonal data and powers operational forecasting in retail, energy, and aviation.

• Gaussian Mixture Models: Customer segmentation (grouping shoppers by purchase patterns), anomaly detection (flagging low-probability points under the mixture), image segmentation (foreground/background separation in computer vision), speech recognition (modeling phoneme distributions — the dominant pre-deep-learning approach), bioinformatics (clustering gene expression data), and as a generative model for synthetic data generation.

• Model comparison and error metrics: A universal skill — every predictive modeling project, from Kaggle competitions to production ML systems, requires choosing between candidate models using quantitative criteria. MSE/MAD/MAPE/LAD are the four lenses through which forecast (and, more broadly, prediction) quality is measured.