Chapter 6 Application (6.4&6.5)

Today, I finished reading Chapter 6 to find something useful for my research.

Summary

Because the dimension reduction offered by PLS is supervised by the response, it is more quickly steered towards the underlying relationship between the predictors and the response.

NIPALS: nonlinear iterative partial least squares algorithm

The constructs of NIPALS could be obtained by working with a “kernel” matrix of dimension P × P, the covariance matrix of the predictors (also of dimension P×P), and the covariance matrix of the predictors and response (of dimension P ×1). This adjustment improved the speed of the algorithm, especially as the number of observations became much larger than the number of predictors.

SIMPLS: simple modification of the PLS algorithm.

If a more intricate relationship between predictors and response exists, then we suggest employing one
of the other techniques rather than trying to improve the performance of PLS through this type of augmentation.

Partial least square regression vs. Ordinary least square regression

Penalized models

Combatting collinearity by using biased models may result in regression models where the overall MSE
is competitive.

One method of creating biased regression models is to add a penalty to the sum of the squared errors.

Ridge regression adds a penalty on the sum of the squared regression parameters:

Penalty increase, bias increase, variance decrease, model under-fit.

Lasso: least absolute shrinkage and selection operator model

A generalization of the lasso model is the elastic net. This model combines the two types of penalties:

The advantage of this model is that it enables effective regularization via the ridge-type penalty with the feature selection quality of the lasso penalty.

Tomorrow, I will continue to read Chapter 7 of the book.