LASSO

LASSO applies the L1-norm instead of the L2-norm as in ridge regression, which is the sum of the absolute value of the feature weights and so minimizes RSS + λ(sum |Bj|). This shrinkage penalty will indeed force a feature weight to zero. This is a clear advantage over ridge regression, as it may improve the model interpretability.

The mathematics behind the reason that the L1-norm allows the weights/coefficients to become zero is beyond the scope of this book (refer to Tibsharini, 1996 for further details).

If LASSO is so great, then ridge regression must be obsolete in machine learning. Not so fast! In a situation of high collinearity or high pairwise correlations, LASSO may force a predictive feature to zero, hence you can lose the predictive ability; that is, if both feature A and B should be in your model, LASSO may shrink one of their coefficients to zero. The following quote sums up this issue nicely:

"One might expect the lasso to perform better in a setting where a relatively small number of predictors have substantial coefficients, and the remaining predictors have coefficients that are very small or that equal zero. Ridge regression will perform better when the response is a function of many predictors, all with coefficients of roughly equal size."

– James, 2013

There is the possibility of achieving the best of both worlds and that leads us to the next topic, elastic net.

Table of Contents for LASSO

Create new playlist

Sign In

Sign Up

Table of Contents for
LASSO