ETH_IntroductionToMachineLearning

links - IntML

course site: https://las.inf.ethz.ch/teaching/introml-s26
projects: https://project.las.ethz.ch/

Info

exam
- multiple choice, fully on paper
projects
- 3/4 have to "pass" to be able to do the exam
- 1min video explaining solution for each project needed, uploaded to polybox
practice classes
- ETH_IntroductionToMachineLearning_Übungen

Hilfsmittel IntroductionToMachineLearning

2 A4 pages (1 sheet) handwritten / >11pt
calculator

Exercises

-> FS2026_tasks

Serie	Lösung	Korrektur

Vorlesung

#timestamp 2026-02-27

#todo completely useless - delete or rewrite

We have:

\begin{aligned} \hat{w} & = w_{s} + w_{p} \\ w_{s} & \in span (X^{⊤}) = span (X^{⊤} X) \end{aligned}

normal equation:

\begin{aligned} X^{⊤} X ω & = X y \\ ⟺ ω & = (X^{⊤} X)^{†} X y = X^{†} y \end{aligned}

#timestamp 2026-03-03

Variance might not be only because of noise, but also because there are not enough samples.

overfitting $\hat{=}$ too sensitive to noise

#timestamp 2026-03-04

Bias $\sim$ distance between average $\overset{―}{f} = \frac{1}{J} \sum_{j = 1}^{J} {\hat{f}}_{D_{j}}$ and $f^{⋆}$
- error from wrong assumptions in algorithm ("underfitting)
Variance $\sim$ how far ${\hat{f}}_{D_{j}}$ is from $\overset{―}{f}$
- error from sensitivity to noise ("overfitting")

-> increasing model complexity beyond point where training error $= 0$ , can lead to second decrease in generalization error (more than just u-curve)

splitting data

Train/Test split: Normally $\frac{9}{10}$ - $\frac{1}{10}$ test split, model is evaluated on test data
Train/Validation/Test split: e.g. $50 % - 25 % - 25 %$ , use validation set to tune hyperparameters, use test set to get unbiased estimate
K-Fold Cross-Validation (CV): Dividing the training data into $K$ subsets (folds). The model is trained $K$ times, each time using a different fold as the validation set and the remaining $K - 1$ folds for training.

control model complexity
Complex model (e.g. high-degree polynomial) might make weights large / osscilate. To counteract:

Smaller degree $m$
Smaller number of monomials “active” by limiting $l_{1}$ -norm
Limit $l_{2}$ norm

Lasso Regression ( $ℓ_{1}$ Regularization)
Adds a penalty proportional to the absolute value of the coefficients.

{\hat{w}}_{lasso} = {argmin}_{w} | | y - Φ w | |_{2}^{2} + λ | | w | |_{1}

Effect: Induces sparsity, meaning it sets some coefficients to exactly zero, effectively performing feature selection.

Ridge Regression ( $ℓ_{2}$ Regularization)
Adds a penalty proportional to the square of the magnitude of coefficients.

{\hat{w}}_{ridge} = {argmin}_{w} | | y - Φ w | |_{2}^{2} + λ | | w | |_{2}^{2}

Effect: Shrinks all coefficients toward zero but rarely makes them exactly zero.
Analytical Solution: ${\hat{w}}_{ridge} = (X^{T} X + λ I)^{- 1} X^{T} y$ .

Why Lasso Induces Sparsity

The geometric intuition lies in the shape of the constraint regions. The $ℓ_{1}$ ball (a diamond/cross-polytope in 2D) has "corners" on the axes. When the elliptical contours of the least-squares loss function expand, they are highly likely to first hit the $ℓ_{1}$ ball at one of these corners, where one or more coordinates are zero.

Comparison Table

Feature	Ridge (ℓ2)	Lasso (ℓ1)
Penalty	$λ \sum w_{j}^{2}$	$\lambda \sum
Solution	Closed-form	Numerical optimization
Sparsity	No (coefficients $\to 0$ )	Yes (coefficients $= 0$ )
Use Case	Preventing overfitting	Feature selection

2026-03-04-3.0f

By the way, to unlock the full functionality of all Apps, enable Gemini Apps Activity.