What do L1 regularization and L2 regularization have in common?

University of Michigan - Ann Arbor

Regularization is a modification made to a learning algorithm with the goal of reducing generalization error, but not training error.

Linear Regression is useful for modeling outputs whose relation to their inputs is linear. However, linear regression performs poorly when trying to model non-linear relationships. We can use regularization to help learning algorithms select certain models over others, particularly to avoid overfitting. 

Regularization

Like ridge and lasso regression, a regularization penalty on the model coefficients can also be applied with logistic regression, and is controlled with the parameter C. 

In fact, the same L2 regularization penalty used for ridge regression is turned on by default for logistic regression with a default value C = 1. With large values of C, logistic regression tries to fit the training data as well as possible. While with small values of C, the model tries harder to find model coefficients that are closer to 0, even if that model fits the training data a little bit worse. 

If the L1 and L2 norms are added to the logistic regression, the effect is: feature selection can be done, and overfitting can also be prevented to a certain extent. Reason: L1–LASSO can generate sparse solutions for feature selection; L2–Ridge constrains model parameters to prevent overfitting. In addition, L2 can get smooth weights.

Logistic Regression - Regularization

While adding an extra term penalizes the weight matrix from becoming too large, why does this prevent overfitting? This is because if lambda becomes very large the weight of the matrices is incentivized to be close to 0. In other words, you are zeroing the impact of hidden units so that a simpler network is the result. The hidden units are still there, but now they have a much smaller effect. With a small W and and small Z, the activation function will likely be fairly linear. 


Why does regularization prevent overfitting? 

- L1 Regularization
- L2 Regularization
- Dropout Regularization
- Data Augmentation
- Early Stopping
- Tangent Distance
- Tangent Prop
- Manifold Tangent Classifier

Popular Regularization Techniques in Deep Learning

You define 0.1% as “human-level performance.” After working further on your algorithm, you end up with the following: 

- Human-level performance: 0.1%
- Training set error: 2.0%
- Dev set error: 2.1%

Human Level Performance: Based on the evidence below, which two of the following four options seem the most promising to try? 

Local constancy and smoothness priors are prior beliefs or assumptions used in machine learning stating that the function being learned should be locally constant or smooth. These priors rely on the assumption that a function should satisfy $$f(x) \approx f(x+\epsilon)$$ for a small $$\epsilon$$. For example, $$k$$-nearest neighbor algorithms directly copy output values from nearby data points, while most kernel methods interpolate between nearby data points. Smoothness priors work best for functions with sufficient examples and low dimensionality. However, more complicated functions can be estimated and generalized effectively by dividing their domain into regions and using underlying assumptions to associate each region. For instance, a function with $$O(2^k)$$ regions can be defined by $$O(k)$$ examples if additional assumptions are used to connect these regions.

Local Constancy and Smoothness Priors

While a parameter norm penalty is one way to regularize parameters to be close to one another, the more popular way is to use constraints: to force sets of parameters to be equal. This method of regularization is often referred to as parameter sharing, because we interpret the various models or model components as sharing a unique set of parameters. A signiﬁcant advantage of parameter sharing over regularizing the parameters to be close (via a norm penalty) is that only a subset of the parameters (the unique set) needs to be stored in memory. In certain models—such as the convolutional neural network—this can lead to signiﬁcant reduction in the memory footprint of the model.



	Parameter sharing: to force sets of parameters to be equal

	Main application: Convolutional Neural Network (CNN)

	Advantages: signiﬁcant reduction in the memory footprint of the model




Parameter Sharing

Parameter tying refers to the fact that the two models, A and B, have a certain correlation. For example, both models are doing a classification task. Based on this prior knowledge, we can assume that the parameters of model A and model B should be similar, and then we can use this information through regularization.

Parameter Tying

MTL acts as a regularizer by introducing an inductive bias. As such, it reduces the risk of overfitting and improves its ability to fit random noise.

MTL as a Regularizer

Classical regularization techniques mitigate overfitting by adding a parameter norm penalty to the loss function during training. This extra penalty term restricts the capacity of the model, thereby keeping the learned model simple and reducing its overall complexity. Different parameter norm penalties lead to different regularization behaviors and preferred solutions.

Parameter Penalties in Classical Regularization

A bigger model can raise variance and make overfitting more likely. However, when the training procedure includes effective regularization, increasing model size often does not harm test performance much. In that case, the biggest practical cost of using the larger model is usually extra computation and memory, not a large drop in generalization.

Learn Before

Related