Exercises

-> FS2026_tasks

Projects

• 20260408_IntML_project2
• 20260426_IntML_project3

Vorlesung

#timestamp 2026-02-27

#todo completely useless - delete or rewrite

We have:

normal equation:

#timestamp 2026-03-03

Variance might not be only because of noise, but also because there are not enough samples.

overfitting $\widehat{=}$ too sensitive to noise

#timestamp 2026-03-04

• Bias $\sim$ distance between average $\bar{f}=\frac{1}{J}\sum _{j=1}^J{\hat{f}}_{D_j}$ and $f^{\star }$
  • error from wrong assumptions in algorithm ("underfitting)
• Variance $\sim$ how far ${\hat{f}}_{D_j}$ is from $\bar{f}$
  • error from sensitivity to noise ("overfitting")

-> increasing model complexity beyond point where training error $=0$, can lead to second decrease in generalization error (more than just u-curve)

splitting data

• Train/Test split: Normally $\frac{9}{10}$ - $\frac{1}{10}$ test split, model is evaluated on test data
• Train/Validation/Test split: e.g. $50\mathrm{\%}-25\mathrm{\%}-25\mathrm{\%}$, use validation set to tune hyperparameters, use test set to get unbiased estimate
• K-Fold Cross-Validation (CV): Dividing the training data into $K$ subsets (folds). The model is trained $K$ times, each time using a different fold as the validation set and the remaining $K-1$ folds for training.

control model complexity
Complex model (e.g. high-degree polynomial) might make weights large / osscilate. To counteract:

1. Smaller degree $m$
2. Smaller number of monomials “active” by limiting $l_1$-norm
3. Limit $l_2$ norm

Lasso Regression (${\ell }_1$ Regularization)
Adds a penalty proportional to the absolute value of the coefficients.

• Effect: Induces sparsity, meaning it sets some coefficients to exactly zero, effectively performing feature selection.

Ridge Regression (${\ell }_2$ Regularization)
Adds a penalty proportional to the square of the magnitude of coefficients.

• Effect: Shrinks all coefficients toward zero but rarely makes them exactly zero.
• Analytical Solution: ${\hat{w}}_{\text{ridge}}=(X^{T}X+\lambda I{)}^{-1}{X}^{T}y$.

#todo why does lasso induce sparsity?

#timestamp 2026-03-20

check whether kernel is valid:

1. check symmetry
2. decompose into known valid kernels (using linearity)
3. use closure rule (sum/product/scaling/composition)
4. if unsure, build a small kernel matrix counterexample

10. Neural networks

#timestamp 2026-03-24

computation at each layer consists of three parts:

• $w$, trainable weights
• $b$, biases, which shift our functions (linear -> affine)
• $\varphi /\phi$, activation functions which scale the values and introduces non-linearity -> allows the network to learn complex patterns
  • possible activation functions: ReLU, Sigmoid

forward propagation

• initial input (input vector $x$ assigned to 0th layer):

• hidden layers can be written as matrix-multiplication (pre-activation value $z$, activation value $h$):

• output layer ($L$): linear combination of last hidden layer activations:

=> for a multilayer perceptron with a single hidden layer:

backwards propagation:
https://xnought.github.io/backprop-explainer/

Dimensionality reduction

PCA
PCA relies on the Covariance Matrix $\mathrm{\Sigma }$. This matrix tells us how every dimension in your data moves in relation to every other dimension.

• The Matrix: $\mathrm{\Sigma }=\frac{1}{n}\sum _{i=1}^n{\mathbf{x}}_i{\mathbf{x}}_i^T$.
• The Magic (Eigenvectors): We solve $\mathrm{\Sigma }\mathbf{v}=\lambda \mathbf{v}$.
  • $\mathbf{v}$ (Eigenvector) is the direction of the "best angle" (the principal component).
  • $\lambda$ (Eigenvalue) is a number representing the amount of spread (variance) in that direction.
• The Projection: To turn your $d$-dimensional point $\mathbf{x}$ into a $k$-dimensional point $\mathbf{z}$, you just multiply it by the top eigenvectors:

kernel PCA
In standard PCA, we find eigenvectors of the data. But what if we first transformed the data into a crazy high-dimensional space using a function $\varphi (\mathbf{x})$? We can't actually compute $\varphi (\mathbf{x})$ because it might be infinite-dimensional.

• The Kernel Trick: Instead of calculating $\varphi (\mathbf{x})$, we use a Kernel Function $k({\mathbf{x}}_i,{\mathbf{x}}_j)$ that calculates the "similarity" between two points directly.
• The Kernel Matrix ($\mathbf{K}$): We build a matrix where every entry $K_{ij}$ is the similarity between point $i$ and point $j$.
• The Solution: We find the eigenvectors $\mathbf{v}$ of $\mathbf{K}$ instead of $\mathrm{\Sigma }$.
• The Projection: The new coordinates $z$ for a point $\mathbf{x}$ are calculated by checking its similarity to all training points:

autoencoders
This is an optimization problem. We have two functions: an Encoder $E(\mathbf{x})$ and a Decoder $D(\mathbf{z})$.

• The Bottleneck: $\mathbf{z}=\sigma ({\mathbf{W}}_{enc}\mathbf{x}+{\mathbf{b}}_{enc})$. Here, $\sigma$ is a nonlinear "gate" (like ReLU or Sigmoid) that allows the math to learn curves instead of just straight lines.
• The Loss Function: We want the output $\hat{\mathbf{x}}$ to be as close to the input $\mathbf{x}$ as possible:

• The Training: gradient descent