Spark MLlib: A Framework for Machine Learning at Scale

16.1 Spark MLlib: Motivation, Uses, and Scale

16.1.1 Why Spark MLlib: Horizontal Scaling of Compute

Spark MLlib earns its place among machine learning offerings such as TensorFlow and scikit-learn because of one core property: it runs on top of a distributed cluster built from commodity hardware, so you can grow your compute power by adding machines rather than by buying a bigger single machine. This idea is called horizontal scaling, and it is the first reason anyone reaches for MLlib.

A single powerful machine — say a high-end gaming PC — is capped at somewhere around 64 to 128 CPU cores. Beyond that, you physically cannot add cores. A Spark cluster has no such wall. If you decide your data is too large or your program is running slowly, you simply add more commodity machines to the cluster and immediately gain their cores. Because MLlib is a core component of Spark, it inherits this scalability for free. Every machine learning workload — training loops, entropy calculations, information-gain computations — runs across however many systems you choose to put in the cluster.

There is also a strategic reason to prefer scaling over optimizing. If you face a choice between making your model run faster on one machine (optimization) or training it on more data across a cluster (scaling), scaling is the better investment. The amount of data generated every year grows exponentially, and pattern detection across a larger dataset almost always beats micro-optimizing a single-node training loop. This is why the course frames MLlib around scale first: your modeling effort is better spent on ingesting more records than on squeezing the last few percent of speed out of local code.

MLlib is available in several languages — Python (as pyspark.ml), Java, and Scala. One point worth clearing up about that list:

16.1.2 Worked Example: Adding Nodes to a Cluster

The scaling argument is easiest to see with numbers. The framing used in the discussion runs like this.

Sense-check: total compute is a product — nodes times cores per node — so doubling the node count doubles the parallelism. A single fixed machine cannot do that at all. The arithmetic is secondary; the shape of the argument is what matters: every added node multiplies capacity, and that is exactly the growth pattern single-machine optimization cannot reproduce.

16.1.3 What You Can Use MLlib For

MLlib serves two distinct roles in a machine learning workflow.

Role 1 — Preprocessing and feature engineering only. When you have a very large dataset, you can use MLlib to clean data, find the most important features among all available independent variables, and run algorithms such as PCA to compress the data. The discussion mentions real systems with 250 feature variables — a huge number — where you may only want the top 10 or 15 features to feed into your algorithm. After MLlib finishes this preprocessing, you can dump the resulting smaller, clean dataset into a traditional ML framework such as scikit-learn and do the actual training there. In this mode, MLlib is used purely for the data-preparation stage.

Role 2 — Full distributed training. If your data is still very large even after preprocessing, MLlib can also perform the real training itself. Spark supports both machine learning (ML) and deep learning (DL) workloads, so you are not forced to move the data anywhere else.

This split matters for planning: preprocessing is where most of the engineering effort lives, and MLlib can serve as the preprocessing engine for any downstream framework.

Real-world: real systems with hundreds of feature variables (one example mentioned 250) use MLlib to clean data and run PCA to narrow down to the top 10–15 features before handing the data to frameworks like scikit-learn for training. This "MLlib as a preprocessing engine" pattern is common in industry precisely because feature engineering is where most of the work lives.

16.1.4 The Spark Component Stack and the DataFrame API

MLlib sits inside a larger Spark ecosystem, and it helps to see where. The stack discussed in the lecture is:

• Spark Core — the base engine that handles how data moves from one system to another when a request for data is made.
• Spark SQL — for processing data with SQL queries.
• Spark Streaming — for processing data as it arrives, before it rests anywhere (stream processing).
• Spark MLlib — the machine learning library, the subject of this lecture.
• GraphX — for modeling social networks and other graph-structured data.

All of these libraries are built on top of the DataFrame APIs. The older RDD API is becoming outdated, and Spark officially deprecates using RDDs for machine learning, so the entire focus here is on the DataFrame-based approach. Even inside MLlib there are two ways to operate — the DataFrame API (with SQL support) and the raw RDD path — but the RDD path is deprecated and not covered.

16.1.5 Scale Limits: Features, Classes, and Training Instances

The scale at which MLlib operates is one of its defining characteristics. The figures below come from the Spark Definitive Guide, the same book that anchors much of the course, and they describe practical limits for popular algorithms:

• Decision Trees — can handle up to 1,000 features. With that many features, choosing the best feature for the root node or each branch node means computing entropy and information gain across thousands of candidates, which is genuinely complex — and Spark does it.
• Random Forest — can handle up to 10,000 features and classification with up to 100,000 (100k) distinct output classes. For example, given a set of features, you could classify which of about 100,000 animals it is.
• Logistic Regression — can go up to 10,000,000 (10 million) features. The spoken figure "up to 10" in the lecture means 10 million; the Spark Definitive Guide confirms that linear models (of which logistic regression is one) handle up to 10 million features.
• Training instances — there is effectively no limit on the number of training examples, for either classification or regression. Spark does not constrain how much data your model can consume to find patterns.

Exam note: Remember the scale figures — 1,000 features for decision trees, 10,000 features and 100,000 output classes for random forests, and a practically unlimited number of training examples. These specific numbers are the kind of detail that appears in exams.

The Spark Definitive Guide is strongly recommended for MLlib. Its authors are the primary authors of Spark themselves, it gives a clean introduction to MLlib, and every example in the book comes with downloadable code and training datasets from a GitHub site. You can replicate the code, plug in the data, and run it directly.

Real-world: Production teams choose Spark MLlib when feature counts reach the thousands and training data reaches billions of rows — volumes that single-node libraries cannot hold in memory. This is why the scale limits in the table are not academic: they define the region of the design space where Spark is the only practical choice.

16.2 The Machine Learning Pipeline in Spark

16.2.1 The Four Stages of a Machine Learning Workflow

Every machine learning program, regardless of framework, passes through the same four stages:

1. Load and clean the data. You load data from a CSV file, a database, HDFS, or another source. Whatever the source, raw data is always impure, so you spend dedicated effort cleaning it.
2. Feature engineering. Here you crunch the features: if you have hundreds of features, you narrow them down to a smaller set, combine features, drop some, find the most important ones, and apply transformations. Somewhere between 60% and 70% of the entire job of machine learning is data preparation; only the last 30% is aided by machine learning algorithms themselves.
3. Model training with hyperparameter tuning. You feed the cleaned feature output into a chosen model. Along with training, you tune hyperparameters — the algorithm's own settings, such as the depth of a decision tree or the minimum number of nodes after which a tree must split. Training typically involves several trials over the same data with different hyperparameter values.
4. Evaluation. You quantify how good the model is with objective measures rather than intuition. You compute a quantitative metric and judge whether the algorithm is good or bad.

16.2.2 Spark's Abstractions for the Pipeline

Spark formalizes these stages with a small set of named abstractions:

• Transformers — used for feature engineering. A transformer takes a DataFrame in and produces a DataFrame out.
• Estimators — used both for feature engineering and for model training. The same word is used for both because both operations learn something from the data before producing a usable object (a transformer or a model).
• Evaluators — used for evaluation. An evaluator gives you a framework where you can pick any metric you care about and get its value. For linear regression, the options include RMSE (root mean squared error), mean squared error, absolute squared error, and \(R^2\). For classification, people discuss the ROC curve, accuracy, specificity, and sensitivity.
• Pipelines — you declare the transformer, the estimator, the modeling algorithm, the parameter list, and the evaluator as separate objects, then tape them all together into a concept called a pipeline. A single execution line then runs every stage in order over the incoming data. scikit-learn has a similar pipeline concept.

The two roles of an estimator are the subtle point: a scaler (feature engineering) and a linear regression (training) are both estimators because both must look at the data before they can produce something usable. Everything downstream of this section rests on that idea.

16.2.3 Package Naming: .mllib versus .ml

When you start coding in the MLlib world, the first thing to notice is the package structure. The original MLlib implementation was built on RDDs, but the DataFrame-based approach came later and is the recommended one. Everything under org.apache.spark.mllib (PySpark: pyspark.mllib) is the RDD-based, deprecated API. Everything under org.apache.spark.ml (PySpark: pyspark.ml) is the DataFrame-based API you should use. Java is very similar to Scala. The lecture focuses only on the .ml DataFrame-based code.

16.2.4 Vectors: Dense and Sparse

In Spark ML, labels and features are given as vectors — a structure much like an array of doubles, but slightly more efficient and built for fast calculations. There are two kinds:

• Dense vector — every position in the array holds a value. A dense vector of size 100 (or 200, or millions) stores values at most or all of its positions. You declare it with vectors.dense(1.0, 2.0, 3.0), which creates a vector of size 3 with values \(1.0, 2.0, 3.0\):

\[ \mathbf{v}_{dense} = [1.0,\ 2.0,\ 3.0] \]

• Sparse vector — only a few positions hold meaningful values, just like a sparse matrix. You declare it by giving three things: the size of the vector, the locations (indices) that contain data, and the values at those locations. For example, a sparse vector of size 3 with value 2.0 at location 1 and value 3.0 at location 2:

\[ \mathbf{v}_{sparse} = \text{vectors.sparse}(3,\ [1,\ 2],\ [2.0,\ 3.0]) \]

In a sparse vector of 1,000 positions with data only at positions 100 and 200, you would give the size 1000, the indices [100, 200], and the values at those indices. Sparse vectors matter later for one-hot encoding, where most entries of the encoding are zero.

The relevant package is pyspark.ml.linalg, and you import Vector from it. The lecture defines dense vectors as vectors.dense(1, 2, 3) and sparse vectors as vectors.sparse(size, ids, values).

16.2.5 Data Types for Supervised, Unsupervised, Recommendation, and Graph Problems

The shape of the input data depends on the kind of problem:

• Supervised learning — you have features and labels. Features are given as a vector of doubles (you do not pass 12 features as 12 separate columns; you bundle them into one vector). The label — the value to be predicted — must be a double. Integers are not supported, and neither are strings or booleans. Spark operates in doubles in most places: a vector of doubles for features and a double for the label.
• Unsupervised learning — there are no labels and nothing to predict; you are only detecting patterns. The input is a vector of doubles for features.
• Recommendation — the data has a column of users, a column of items (the things being recommended), and a column of rewards or ratings. A recommendation system trains on these three columns.
• Graph analytics — you supply a DataFrame of vertices and a DataFrame of edges, and these go into the training.

16.2.6 A First Sample Program

The pattern to absorb is the two-phase flow: fit happens only on training data, and only the resulting model is allowed to touch the test data. That separation is what keeps the evaluation honest — the model never sees the test answers while learning. This first example deliberately skips feature engineering and says nothing about what is inside the data. Raw data always needs cleaning, and once you understand how the cleaning and transformation pieces work, the fit and transform pattern becomes easy to follow.

Real-world: The train/test split with a fixed random seed (here 42) is standard practice everywhere, because reproducible splits let multiple engineers compare models on exactly the same held-out data.

16.3 Transformers and Estimators: The Core of Spark ML

16.3.1 Transformer Mechanics

A transformer takes a DataFrame as input and outputs a DataFrame. Usually it does some processing on one column and creates one more column as a result. Spark mostly operates on immutable data — it does not like modifying data in place, because modifying distributed data in the middle of a cluster is a complicated operation. So instead of editing an existing column, a transformer appends a brand-new column to the DataFrame. If you start with 10 columns and process one of them, you end with 11 columns, where the 11th is the transformed version of the column you specified.

To use a transformer you must tell it which column to operate on. Every transformer has a function setInputCol (and in some cases setInputCols for multiple input columns). Setting the input column is mandatory. The output column is controlled by setOutputCol; this is optional, and if you omit it Spark generates a column name for you automatically.

Different transformers also have transformer-specific parameters, set through set methods (in Python you can instead pass keyword arguments at construction time). For example, a min-max scaler needs minimum and maximum values; a word-splitting transformer does not. After configuring the object, you call one method — transform — passing the data as an argument.

A concrete example is the Tokenizer, which splits a sentence into words. In Scala:

val tokenizer = new Tokenizer()
  .setInputCol("description")
  .setOutputCol("words")
val tokenized = tokenizer.transform(sales.select("description"))

You create the tokenizer object, set which column to transform (description), set the output column, and call transform. The transformer only ever touches the description column, even if you pass it the entire DataFrame.

16.3.2 Estimator Mechanics: The Two-Pass Flow

An estimator also transforms data, but it cannot work on its own — it must go through the data at least once before it can transform anything. The example used to explain this is min-max scaling. To scale data you give two arguments: the new minimum and the new maximum (say 0 and 10). But the algorithm cannot do anything with just those inputs. It must first scan the data to discover what the current global minimum and global maximum are. If the data ranges from -10 to 10, it needs to know both -10 and 10 before it can map that range onto 0 to 10.

So an estimator performs two passes:

1. First pass — fit. You call fit with the data. The estimator scans the data once, computes the required metadata (for min-max scaling, the global min and max), and returns a transformer object that carries those learned values.
2. Second pass — transform. You call transform on the returned transformer, passing the same data, and get the final transformed output.

In Scala:

val scaler = new StandardScaler()
  .setInputCol("features")
  .setOutputCol("scaledFeatures")
val scalerModel = scaler.fit(scaleDF)     // pass 1: learn mean and standard deviation
val scaled = scalerModel.transform(scaleDF) // pass 2: apply the learned statistics

The pattern is always: create the estimator object, set the input column (and any estimator-specific parameters), call fit on the data to get a transformer, then call transform on that transformer with the data.

16.3.3 The Distinction in One Line

If a transformation contains all the details it needs, you apply it directly as a transformer. If the transformation depends on statistics that must be discovered from the data first, you go through an estimator: fit to discover the metadata, then transform to apply it.

When to pick which: if the operation is a fixed rule that needs no dataset-wide statistics, use a transformer; if it must discover something from the data first (min, max, mean, standard deviation, vocabulary, category list), use an estimator.

Real-world: In production ML pipelines, choosing transformer vs estimator correctly determines whether a stage is reproducible and how it behaves on new data. A misclassified estimator (for example, computing min-max bounds from only a sample) silently corrupts every downstream prediction, which is why the fit/transform split is enforced by the API itself — and why libraries like scikit-learn use the identical two-phase design.

16.4 Feature Engineering Toolkit: Transformers

16.4.1 VectorAssembler: Combining Columns into One Vector

Spark ML expects all your features to be available in the form of a single vector, unlike scikit-learn, where you hand each column separately. So if your DataFrame holds features in separate columns (in1, in2, in3), you must combine them into one vector column before the model will accept them.

The VectorAssembler does exactly this. It is a transformer — it does not need to scan the data to combine columns; you give it three columns and it knows how to put them together. You set the input columns with setInputCols(Array("in1", "in2", "in3")) and call transform on the data. The result shows the three values for each sample packed into a single vector. This is the most basic and most frequently used transformer, because DataFrame columns eventually have to become a vector to reach the algorithm.

16.4.2 Bucketizer: Turning Continuous Ranges into Buckets

Continuous values with many distinct levels are often not useful as-is. Age is the classic case: a feature variable with a range from 0 to 100 has enormous cardinality, but you may prefer to classify people into a few meaningful groups — infants, toddlers, teenagers, adults, and senior citizens. Similarly, you might classify weight into underweight, average, and overweight. Bucketing replaces a continuous value with the number of the bucket it falls into.

The Bucketizer does this. It is a transformer. You set the input column (the lecture example uses a column called id), and you set the bucket boundaries with the transformer-specific setSplits method. In the example:

new Bucketizer()
  .setInputCol("id")
  .setSplits(Array(-1.0, 5.0, 10.0, 250.0, 260.0))

This creates four buckets: from -1 to 5, from 5 to 10, from 10 to 250, and from 250 to 260. An id of 0 falls into the first bucket (output 0); an id of 10 falls into the third bucket (output 2, covering 10 to 250). To have \(n\) buckets you need \(n+1\) boundaries; the setSplits array must have one more value than the number of buckets, and the Bucketizer takes care of the counting.

One detail worth remembering: because no setOutputCol was given in the example, the output column got an automatically generated, awkward name (something like a random suffix appended to bucket). If you set setOutputCol("id_bucket"), you control the name and avoid the odd auto-generated one.

16.4.3 ElementwiseProduct: Scaling Each Feature by a Factor

When you want to multiply a feature vector element-by-element by a scaling vector, you use the ElementwiseProduct. Where a dot product is a sum of products, an elementwise product is just the products — each position in the feature vector is multiplied by the matching position in the scaling vector.

In the example, the features column holds three values (1, 0.1, -1) in the first row, and the scaling vector is a dense vector (10, 15, 20). Since elementwise multiplication requires both vectors to have the same length, a three-element feature vector pairs with a three-element scaling vector:

\[ (1,\ 0.1,\ -1) \odot (10,\ 15,\ 20) = (1 \times 10,\ 0.1 \times 15,\ -1 \times 20) = (10,\ 1.5,\ -20) \]

where \(\odot\) denotes elementwise (Hadamard) multiplication. The scaling vector is created as vectors.dense(10, 15, 20), the input column is set to features, and transform produces the multiplied result. The ElementwiseProduct is a transformer — no data scan is needed.

16.4.4 IndexToString: Converting Numbers Back to Strings

The StringIndexer (covered in the estimators section) converts categorical strings to numbers in the forward direction. When a model predicts a numeric label, you usually want to convert it back to a readable string — a predicted 0 means nothing to a human, but "bad" does. That reverse conversion is done by IndexToString.

An important correction: to go forward (string to number) you need an estimator, because the set of unique categories must be discovered. To go backward (number to string) you do not need an estimator. Spark stores the mapping — "good means 1.0, bad means 0.0" — as metadata on the column during the forward conversion. The reverse transformer reuses that metadata, so it never needs to rescan the data, and it cannot assign a random label; it must return what was established earlier. So IndexToString is a transformer, not an estimator.

Usage is symmetric: setInputCol("labelIndicator"), then transform the data. The forward pass turned "good" into 1.0 and "bad" into 0.0; the reverse pass turns 1.0 back into "good" and 0.0 back into "bad".

16.4.5 SQLTransformer: Transformations as SQL

If none of the ready-made transformers fit your need and SQL feels natural, Spark provides the SQLTransformer. You write the actual query and pass it as a statement. The query's table name is always the keyword __THIS__, which stands for the DataFrame being passed in.

For example:

new SQLTransformer()
  .setStatement("SELECT sum(quantity), count(*), customerID FROM __THIS__")

When you call transform(sales), the query effectively runs as SELECT sum(quantity), count(*), customerID FROM sales, where sales is the input DataFrame. In Spark SQL you normally register a temp table and query it; here the table name is always __THIS__. The SQLTransformer is a single transformer — it is not an estimator.

16.4.6 Tokenizer: Splitting Text into Words

The Tokenizer breaks a piece of text into individual words. It is a pure transformer because splitting on whitespace needs no prior scan of the data. You set the input column (for example description), set the output column (for example words), and call transform. This is the same tokenizer used in the transformer introduction: new Tokenizer().setInputCol("description").setOutputCol("words").

The tokenizer is the entry point of the NLP chain: raw text enters, a list of words exits, and every later text transformer (StopWordsRemover, NGram, CountVectorizer) consumes that list.

16.4.7 StopWordsRemover: Dropping Frequent, Meaningless Words

In text processing you often want to remove stop words — frequently used words that carry little meaning. In the sentence "the BDA class is interesting", the words "the" and "is" do not help decide whether the sentence is positive or negative, while "class" and "BDA" do.

The StopWordsRemover removes these words using a predetermined list per language. The languages supported include Danish, Dutch, English, Finnish, Turkish, and others. You choose the language, get the stop-word list, set the input column with setInputCol, and call transform on the data. Starting from the tokenizer's output, the stop-word remover drops words such as "off" when they appear on the stop list.

16.4.8 NGram: Word Collocations

The NGram transformer helps you understand which words frequently appear together — their collocation. Given the sentence "for understanding the collocation of words":

• A 1-gram is each single word: "for", "understanding", "the", "collocation", "of", "words".
• A 2-gram (bigram) is each pair of adjacent words: "for understanding", "understanding the", "the collocation", "collocation of", "of words".
• A 4-gram is each window of four adjacent words: "for understanding the collocation", "understanding the collocation of", "the collocation of words".

You pass an argument, say \(n\), and the NGram transformer breaks the text into that many-word units. This is very specific to NLP; the first pages of any NLP course introduce tokenizers and n-grams. Like the tokenizer, NGram is a transformer.

Real-world: N-grams are a standard building block for language models and autocomplete systems, where predicting the next word depends on the previous one or two words. The transformer chain — tokenize, remove stop words, build n-grams, vectorize — is the backbone of production text analytics pipelines on Spark.

16.5 Feature Engineering Toolkit: Estimators

16.5.1 QuantileDiscretizer: Bucketing by Percentiles

The QuantileDiscretizer buckets continuous values using quantiles — the value below which a given percentage of the data falls. If you ask for the 70th percentile, you mean the value such that 70% of the data lies below it. You set the number of buckets with setNumBuckets; asking for 5 buckets means the discretizer finds the 20th, 40th, 60th, 80th, and 100th percentiles and splits the data there.

The QuantileDiscretizer is an estimator, not a transformer, because the percentile boundaries depend on the data. Consider 200 numbers with values 1 through 200. The 50th percentile is 100 — half the data lies below 100 and half above. If the same range instead holds 1,000 numbers, the 50th percentile becomes 500. The boundary moves with the data, so the algorithm must scan the data once to find where to break it before it can bucket anything.

The pattern follows the estimator flow: setInputCol on the column to operate on, setNumBuckets(5) as the estimator-specific parameter, then bucketer.fit(data) to discover the boundaries (returning a transformer), and finally transform(data) on that transformer to assign bucket numbers.

16.5.2 StandardScaler: Z-Score Normalization

Normalization here means z-scoring: each value is replaced by how many standard deviations it sits from the mean. The new value is:

\[ z = \frac{x - \mu}{\sigma} \]

where \(x\) is the original variable, \(\mu\) (mu) is the mean of the data, and \(\sigma\) (sigma) is the standard deviation. If the values are 0 and 10, the mean is \(\mu = 5\) and the standard deviation is \(\sigma = 5\), so:

\[ z_0 = \frac{0 - 5}{5} = -1, \qquad z_{10} = \frac{10 - 5}{5} = 1 \]

The StandardScaler performs this transformation, and it is an estimator because it must scan the data at least once to compute the mean and the standard deviation. You set the input column with setInputCol("features"), call scaler.fit(data) to learn \(\mu\) and \(\sigma\), and then transform to apply the z-score. A correctly z-scored column shows values in roughly the -3 to +3 range, since nearly all data sits within three standard deviations of the mean.

16.5.3 MinMaxScaler: Rescaling to a Fixed Range

Min-max scaling maps the data's current range onto a new, fixed range. The discussion uses this example more than once. You provide two arguments — the new minimum and the new maximum (for instance 0 and 10) — but the scaler still cannot work until it knows the data's own global minimum and maximum. With data ranging from -10 to 10, the scaler must discover both -10 and 10 before it can map that range onto 0 to 10.

The transformation is:

\[ x' = \frac{x - x_{\min}}{x_{\max} - x_{\min}} \cdot (new_{\max} - new_{\min}) + new_{\min} \]

where \(x_{\min}\) and \(x_{\max}\) are the global minimum and maximum of the data, \(new_{\min}\) and \(new_{\max}\) are the target range (0 and 10 in the example), and \(x'\) is the scaled value. Because the global extremes must be discovered from the data, the MinMaxScaler is an estimator: create the object, set the new min and max (the source example sets them to 5 and 10), set the input column, call fit, then transform.

16.5.4 StringIndexer: Converting Categories to Numbers

Machine learning algorithms cannot work on strings, so categorical variables must be converted to numbers. A categorical column with values "good" and "bad" becomes 1.0 and 0.0. Spark does not allow integers here — the conversion must produce doubles, which is why you see 1.0 and 0.0 rather than 1 and 0.

The StringIndexer performs this conversion, and it is an estimator: it must scan the column once to discover the unique categories (good, bad, average, best, very poor — whatever the cardinality is), assign each a number, and then go through the data again to write the assigned values. You set the input column with setInputCol("lab") and the output column with setOutputCol("labelIndicator") (unlike earlier examples that left the output column auto-named). Then labelIndexer.fit(data) does the discovery pass, and transform(data) writes the numbers.

16.5.5 OneHotEncoder: Categorical Data as Sparse Vectors

Plain integer labels for categories carry a problem: the numbers imply an ordering that does not exist. If green is 1, blue is 2, and red is 3, what does the difference between 2 and 1 mean? Nothing. For pure categories, the standard technique is one-hot encoding, where each category becomes a binary vector with a single position turned on. In the example, green is encoded as (0, 1, 0), red as (1, 0, 0), and blue as (0, 0, 0).

The OneHotEncoder is an estimator because it must scan the column to find all unique categories before it can build the encoding. You set the input column and output column — the code example encodes the already-indexed colorIndex column into a colorIndicator column. The encoder output does not appear as separate columns; it appears as a sparse vector for space efficiency. For a three-category variable, the sparse representation records the size (3, i.e., positions 0, 1, 2) and the single position that holds a 1.0. For example, a category whose 1.0 sits at position 1 is stored as a sparse vector with value 1.0 at index 1 and zeros elsewhere:

\[ \text{vectors.sparse}(3,\ [1],\ [1.0]) \quad \leftrightarrow \quad (0,\ 1.0,\ 0) \]

One-hot encoding matters because it is genuinely one-hot but stored sparsely — the lecture notes explicitly that this is "not actually one-hot encoding [as separate columns], but it is represented as a sparse vector for the space efficiency."

Real-world: One-hot encoding a high-cardinality categorical feature (say 100,000 categories) produces a vector with mostly zeros, which is exactly why Spark stores it as a sparse vector — memory stays manageable.

16.5.6 CountVectorizer: From Words to Numbers

To feed text to a machine learning algorithm, you convert words to numbers. In the example, the corpus of words maps apple to 1, "is" to 2, and red to 3, so the sentence "apple is red" becomes the numbers 1, 2, 3. Because you cannot pass strings to an ML algorithm, this conversion is essential.

The CountVectorizer is an estimator: it must see the corpus of words once to build the vocabulary (which word gets which number) before it can convert any sentence. This follows the same estimator pattern — fit to learn the vocabulary, then transform to convert.

16.5.7 PCA: Finding the Most Important Features

Principal Component Analysis (PCA) finds the most important features in a dataset. The motivating analogy: if someone asks your net worth, you mentally add up your significant assets — market value of assets, bank fixed deposits, forex holdings, land value, jewelry. You would not bother adding the few tens of rupees in your wallet, because your total assets are so large that the wallet amount changes nothing. PCA works the same way: among many features, it identifies the ones that contribute the most, and the near-irrelevant ones do not count.

You use a PCA transformer, set the number of important components you want with setK, and it returns the top features. In the example, the dataset has three features and setK(2) produces the top two features most useful for that dataset. PCA is also one of the algorithms you can use with MLlib purely for preprocessing, before handing the compressed data to a traditional framework.

Real-world: PCA on MLlib is a common preprocessing step in industry — real systems with hundreds of feature variables use it to compress data to a handful of components before training, which is exactly the "MLlib as a preprocessing engine" role from section 16.1.

16.6 Model Training, Hyperparameter Tuning, and Evaluation

16.6.1 The Modeling Steps: fit, transform, evaluate

Once feature engineering is done, modeling follows a fixed sequence:

1. Create the model object of your choice — a linear regression, a random forest regression, or any other algorithm.
2. Fit the training data onto the model with fit. This is where actual training begins. This is exactly why models are called estimators: you pass the training data to fit, and the returned object is the trained model.
3. Run the test data through the trained model to get predictions — you use transform (or evaluate in the linear regression example) on the held-out test data, producing a DataFrame with predicted values.
4. Evaluate the results with an evaluator to quantify how effective the model is.

So the same estimator vocabulary from feature engineering carries over: fit on training data produces a model, and that model object is then used to predict on test data.

16.6.2 Worked Example: Linear Regression on Study Time vs. Grades

The full worked example is a linear regression between study time and grades, based on a small dataset with a clear linear relationship: the more you study, the better the grades.

The pattern to absorb: create the model, fit with training data to get the model and its parameters, evaluate on test data to get predictions, then pass predictions and actuals to an evaluator to get effectiveness numbers.

16.6.3 Hyperparameter Tuning with ParamGridBuilder

Instead of re-running the algorithm by hand for each hyperparameter combination, Spark provides the ParamGridBuilder. You build an object, pass the parameter name and the list of values you want to try, and Spark automatically tries every combination.

In the example:

• addGrid(lr.elasticNetParam, [0, 0.5, 1]) — try the elasticNetParam hyperparameter (one of the values surfaced by lr.explainParams) at three values.
• addGrid(lr.regParam, [value1, value2]) — try the regularization parameter at two values.

With 3 values for one parameter and 2 for the other, Spark implicitly tries \(3 \times 2 = 6\) variants of the program and performs a final evaluation, then reports which hyperparameter combination performed best. You get the comparison for inspection without writing the training loop six times.

16.6.4 Classification: Random Forest and Naive Bayes

Classification follows the same structure as regression, using the two algorithms covered in an earlier discussion — random forest and Naive Bayes. You create a RandomForestClassifier or a NaiveBayes object, inspect the tunable parameters with nb.explainParams() or rf.explainParams(), and train with object.fit(...) — the call that starts the actual training.

For evaluation, classification uses transform on the test data to get predicted labels, and the popular comparison metric is the ROC curve. For binary classification, Spark provides a BinaryClassificationEvaluator, and the obvious metric from it is the area under the ROC curve (areaUnderROC).

By the end, the general structure of an ML program in Spark should be clear, and the applicability of transformers and estimators should carry over: whether you meet a new transformer or a new algorithm later, it will follow the same shape — configure the object, fit on training data, transform or evaluate on test data.

Real-world: This fixed pipeline shape — vector-assemble features, split, fit, evaluate — is what makes Spark ML projects maintainable in industry. New team members can read any model code and immediately find the training, prediction, and evaluation pieces.

Exam Guidance Summary

Study note: Beyond the examination, this material — and machine learning on Spark generally — is worth learning far more deeply for professional life than what any exam requires. The Spark Definitive Guide is the recommended companion, with downloadable examples and training data from GitHub.

Study note: A follow-up session on recommendation systems and graph analytics may be scheduled after the examinations (for example, around December 11th) if there is enough interest.

Key Industry Applications

Real-world: Horizontal scaling with commodity hardware — teams grow compute by adding nodes to a Spark cluster rather than buying larger single machines, which is how ML training keeps pace with exponentially growing data volumes.

Real-world: MLlib as a preprocessing engine — real systems with hundreds of feature variables (one example mentioned 250) use MLlib to clean data and run PCA to narrow down to the top 10–15 features before handing the data to frameworks like scikit-learn for training.

Real-world: MLlib for full-scale training — when the data is still large after preprocessing, MLlib itself performs distributed training, supporting both machine learning and deep learning workloads.

Real-world: Sparse vector storage for one-hot encoding — high-cardinality categorical features produce mostly-zero vectors, and Spark stores one-hot encodings as sparse vectors to keep memory usage practical.

Real-world: The Spark Definitive Guide — written by Spark's primary authors, it is the recommended reference for MLlib and comes with runnable examples and downloadable training datasets from GitHub, the same pattern used in the lecture's worked examples.