Chapter 7 application (7.1)

Today, I read Chapter 7 (7.1) to find something useful for my research.

Summary

Nonlinear Regression Models

Neural Networks

Weight decay is an approach to moderate over-fitting. It is a penalization method to regularize the model similar to ridge regression.

As the regularization value λ increases, the fitted model becomes more smooth and less likely to over-fit the training set.

Of course, the value of this parameter must be specified and, along with the number of hidden units, is a tuning parameter for the model. Reasonable values of λ range between 0 and 0.1. Also note that since the regression coefficients are being summed, they should be on the same scale; hence the predictors should
be centered and scaled prior to modeling.

Neural networks are models which are more accurate than linear models. It can be used to be applied to the log data. After building the data matrix, we may have dozens of predictors, namely log data to predict outcomes, namely NMR data. Before building the model , we can first center and scale predictors and use PCA to decrease correlations between every pair of predictors. Then we can set a set of tuning parameters of weight decay and number of hidden units to build the model. After running the codes, we can find the optimized tuning parameter so that  we can choose the optimized neural network model. It may perform well in predicting NMR data.

If we can just find the local optimization but not the global one, we can use the model averaged neural networks to solve the problem. The approach is that we start do modeling with several different starting numbers so that we can get different results of the model. Then we average them to get the more reliable results.

The following is an example of the comparison of neural networks with different tuning parameters.

Tomorrow, I will continue to read the book.