Regularization is a collection of techniques that can be used to prevent over-fitting. Regularization adds information to a problem, often in the form of a penalty against complexity, to a problem. Occam's razor states that a hypothesis with the fewest assumptions is the best. Accordingly, regularization attempts to find the simplest model that explains the data.

scikit-learn provides several regularized linear regression models. Ridge regression, also known as Tikhonov regularization, penalizes model parameters that become too large. Ridge regression modifies the residual sum of the squares cost function by adding the L2 norm of the coefficients, as follows:

is a hyperparameter that controls the strength of the penalty. Hyperparameters are parameters of the model that are not learned automatically and must be set manually. As increases, the penalty increases, and the value of the cost function increases. When is equal to zero, ridge regression is equal to linear regression.

scikit-learn also provides an implementation of the Least Absolute Shrinkage and Selection Operator (LASSO). LASSO penalizes the coefficients by adding their L1 norm to the cost function, as follows:

The LASSO produces sparse parameters; most of the coefficients will become zero, and the model will depend on a small subset of the features. In contrast, ridge regression produces models in which most parameters are small but nonzero. When explanatory variables are correlated, the LASSO will shrink the coefficients of one variable toward zero. Ridge regression will shrink them more uniformly. Finally, scikit-learn provides an implementation of elastic net regularization, which linearly combines the L1 and L2 penalties used by the LASSO and ridge regression. That is, the LASSO and ridge regression are both special cases of the elastic net method in which the hyperparameter for either the L1 or L2 penalty is equal to zero.