Bias-Variance Tradeoff in Machine Learning

May 3, 2025

A rigorous mathematical decomposition of prediction error into bias, variance, and irreducible noise—with practical intuitions on how to balance them for better-generalizing models.

In this blog post, we explore the fundamental concept of the bias-variance tradeoff in machine learning—an essential idea that helps us understand and manage the performance of predictive models.

1. Mathematical Formulation

Setup

Let $y = f (x) + ϵ$ , where $f (x)$ is the true function, and $ϵ$ is random noise with $E [ϵ] = 0$ and $Var (ϵ) = σ^{2}$ .
Let $\hat{f} (x; D)$ be the model’s prediction for input $x$ , trained on dataset $D$ .
The expected error is defined as the mean squared error over all possible datasets $D$ and noise $ϵ$ :
$Error (x) = E [(y - \hat{f} (x; D))^{2}]$

Derivation

Start with the expected error:

Error (x) = E [(y - \hat{f} (x; D))^{2}]

Substitute $y = f (x) + ϵ$ :

y - \hat{f} (x; D) = (f (x) + ϵ) - \hat{f} (x; D)

Square the difference:

(y - \hat{f} (x; D))^{2} = ((f (x) - \hat{f} (x; D)) + ϵ)^{2}

Expand the squared term:

= (f (x) - \hat{f} (x; D))^{2} + 2 (f (x) - \hat{f} (x; D)) ϵ + ϵ^{2}

Take the expectation over $ϵ$ and $D$ :

Error (x) = E [(f (x) - \hat{f} (x; D))^{2}] + 2 E [(f (x) - \hat{f} (x; D)) ϵ] + E [ϵ^{2}]

Let’s analyze each term:

1. Model Error Term

To simplify the first term, we add and subtract the expected prediction:

f (x) - \hat{f} (x; D) = (f (x) - E [\hat{f} (x; D)]) + (E [\hat{f} (x; D)] - \hat{f} (x; D))

Then square and take the expectation:

E [(f (x) - \hat{f} (x; D))^{2}] = E [(f (x) - E [\hat{f} (x; D)])^{2}] + E [(E [\hat{f} (x; D)] - \hat{f} (x; D))^{2}] + 2 E [(f (x) - E [\hat{f} (x; D)]) (E [\hat{f} (x; D)] - \hat{f} (x; D))]

Bias squared: $Bias^{2} = (f (x) - E [\hat{f} (x; D)])^{2}$
Variance: $Variance = E [(\hat{f} (x; D) - E [\hat{f} (x; D)])^{2}]$

The cross term vanishes due to the independence and linearity of expectation.

E [(E [\hat{f} (x; D)] - \hat{f} (x; D))] = E [\hat{f} (x; D)] - E [\hat{f} (x; D)] = 0

2. Cross Term

2 E [(f (x) - \hat{f} (x; D)) ϵ] = 0

Since $ϵ$ is independent of $\hat{f} (x; D)$ , and $E [ϵ] = 0$ , this term becomes zero.

3. Noise Term

E [ϵ^{2}] = Var (ϵ) = σ^{2}

This is the irreducible error.

Final Formula

Combining the terms, we get the full decomposition:

Error (x) = Bias^{2} + Variance + σ^{2}

Explicitly,

Error (x) = (E [\hat{f} (x; D)] - f (x))^{2} + E [(\hat{f} (x; D) - E [\hat{f} (x; D)])^{2}] + σ^{2}

This expression highlights how total error results from model bias (model inaccuracy), variance (model instability), and inherent noise in the data.

2. What is the Bias-Variance Tradeoff?

The bias-variance tradeoff describes the fundamental tension in model performance: as you reduce bias, variance often increases, and vice versa.

Bias

Definition: Error due to incorrect assumptions in the learning algorithm.
Analogy: A student who skims the textbook and misunderstands the key concepts.
Impact: High bias results in underfitting—the model is too simple to represent the data well.
Example: Using only the number of bedrooms to predict house prices, ignoring other features like location or size.

Variance

Definition: Error from excessive sensitivity to the training data.
Analogy: A student who memorizes practice questions but struggles with new problems.
Impact: High variance leads to overfitting—the model performs well on training data but poorly on unseen data.
Example: A model that captures every noise pattern in the training set but fails to generalize.

Tradeoff Summary

High bias, low variance: The model is too simple (underfitting).
Low bias, high variance: The model is too complex (overfitting).
Low bias, low variance: The ideal model—accurate and stable.

Think of it like tuning a guitar:

Too tight (high variance) or too loose (high bias) both sound wrong.
The right balance is needed for harmony.

Dartboard Analogy

Low bias, low variance: Darts cluster at the bullseye (accurate and precise) -> The model is consistent and accurate.
High bias, low variance: Darts cluster away from the bullseye (consistently wrong) -> The model is consistent but inaccurate.
Low bias, high variance: Darts are scattered but centered around the bullseye (inconsistent) -> The model is uncertain but accurate.
High bias, high variance: Darts scatter far and wide (inaccurate and inconsistent) -> The model is both uncertain and inaccurate.

Dartboard Analogy

3. How to Balance Bias and Variance?

Balancing bias and variance requires careful tuning of model complexity and learning strategy. Some general guidelines include:

Use simpler models (like linear regression or decision stumps) when data is limited or noise is high.
Use complex models (like deep neural networks) when you have large, high-quality datasets.
Regularization techniques (like Lasso, Ridge, Dropout) help prevent overfitting by reducing variance.
Cross-validation allows you to estimate generalization performance and choose the model with the optimal tradeoff.
Ensemble methods (like bagging or boosting) can reduce variance without increasing bias.

Tradeoff

Ultimately, the bias-variance tradeoff is not something to be eliminated, but rather to be managed. The goal is to find the "sweet spot" where your model captures the true signal without being overwhelmed by noise.