Chapter 8 (8.1 8.2)

Today, I read Chapter 8 (8.1 8.2).

Summary
Once the tree has been finalized, we begin to assess the relative importance of the predictors to the outcome.
If SSE is the optimization criteria, then the reduction in the SSE for the training set is aggregated for each predictor.
The model tends to rely more on continuous predictors than the binary ones.
Unbiased Regression Tree Techniques:
GUIDE: generalized, unbiased, interaction detection and estimation algorithm (decouple the process of selecting the split variable and the split value)
Conditional Inference Trees: statistical hypothesis tests are used to do an exhaustive search across the predictors and their possible split points.
For a candidate split, a statistical test is used to evaluate the difference between the means of the two groups created by the split and a p-value can be computed for the test.

Regression Model Trees
M5
The splitting criterion is different.
The terminal nodes predict the outcome using a linear model (as opposed to the single average).
When a sample is predicted, it is often a combination of the predictions from different models along the same path through the tree.
The main implementation of this technique is a “rational reconstruction” of this model call M5.
Split criterion:
reduction=SD(S)-∑_(i=1)^P▒〖n_i/n×SD(S_i)〗

n_i is the number of samples in partition i.
The split that is associated with the largest reduction in error is chosen and a linear model is created within the partitions using the split variable in the model.
Once the complete set of linear models have been created, each undergoes a simplification procedure to potentially drop some of the terms.
Adjusted Error Rate=n^*+pn*-pi=1n*yi-yi

n^* is the number of training set data points that were used to build the model and p is the number of parameters.
Model trees also incorporate a type of smoothing to decrease the potential for over-fitting. The technique is based on the “recursive shrinking” methodology.
The two predictions are combined using
y_((p) )=(n_((k)) y ̂_((k) )+cy ̂_((p) ))/(n_((k))+c)
(the equation on page 185 may be wrong)

y ̂_((k) ) is the prediction from the child node, n_((k)) is the number of training set data points in the child node, y ̂_((p) ) is the prediction from the parent node, and c is a constant with a default value of 15.
Pruning & Smoothing
Smoothing the models has the effect of minimizing the collinearity issues.

Tomorrow, I will continue to read Chapter 8.

Chapter 7 exercises

Today, I finished exercises in Chapter 7 and started reading Chapter 8.

Exercises in Chapter 7
#7.1
set.seed(100)
x=runif(100, min=2, max=10)
y=sin(x)+rnorm(length(x))*0.25
sindata=data.frame(x=x, y=y)
plot(x, y)
datagrid=data.frame(x=seq(2, 10, length=100))
library(kernlab)
rbfsvm=ksvm(x=x, y=y, data=sindata, kernel="rbfdot", kpar="automatic", C=1, epsilon=0.1)
#C: cost of constraints violation; epsilon: epsilon in the insensitive-loss function
#kpar=list(sigma=n), when n gets smaller, the line gets flatter. It can be kpar="automatic".
modelprediction=predict(rbfsvm, newdata=datagrid)
points(x=datagrid$x, y=modelprediction[,1], type="l", col="blue")
#type="p, l, b, c, o, s, h, n"

#7.2
library(mlbench)
set.seed(200)
#inputs are 10 independent variables uniformly distributed on the interval [0,1],
#only 5 out of 10 are actually used in the formula
trainingdata2=mlbench.friedman1(200, sd=1)
#convert 'x' data from a matrix to a data frame
trainingdata2$x=data.frame(trainingdata2$x)
featurePlot(trainingdata2$x, trainingdata2$y)
library(caret)
knnmodel=train(trainingdata2$x, trainingdata2$y, method = "knn", preProcess = c("center", "scale"), tuneLength = 10)
knnmodel
knnpred=predict(knnmodel, newdata=trainingdata2$x)
postResample(pred = knnpred, obs = trainingdata2$y)

#7.3
#build SVM, neural network, MARS and KNN mdoels

# 7.4
#if nonlinear models outperform the optimal linear model, it shows that there is a nonlinear relationship between
#predictors and outcomes. if there is mo significant outperformance, linear model will be recommended.

#7.5
#top important predictors of optimal linear model and optimal nonlinear model are different.
#top predictors which are unique to the optimal nonlinear model is proved to own definitely a nonlinear
#relationship with outcomes

Chapter 8
Regression Trees and Rule-Based Models
Weaknesses: 1. Model instability; 2. Less-than-optimal predictive performance
If the relationship between predictors and the response cannot be adequately defined by rectangular subspaces of the predictors, then tree-based or rule-based models will have larger prediction error than other kinds of models.

8.1
Basic Regression Trees
The predictor to split on and value of the split
The depth or complexity of the tree
The prediction equation in the terminal nodes
CART: classification and regression tree methodology
SSE finds the optimal split-point for every predictor
SSE=∑_(i∈S_1)▒〖(y_i-y ̅_1)〗^2 +∑_(i∈S_2)▒〖(y_i-y ̅_2)〗^2
Cost-complexity tuning
SSEC_p=SSE+Cp×(#Terminal Nodes)
C_p is the complexity parameter
Smaller penalties tend to produce more complex models, which result in larger trees.

Tomorrow, I will continue to read Chapter 8.

Chapter 7 Computing

Today, I finished reading the first four parts of Chapter 7 and completed the computing part of it.

Computing:
library(caret)
library(earth)
library(kernlab)
library(nnet)
#R has a number of packages and functions for creating neural networks such as nnet, neural and RSNNS.
#1. neural networks
nnetfit=nnet(predictors, outcome, size=5, decay=0.01, linout = TRUE, trace = FALSE,
             maxit = 500, MaxNWts = 5*(ncol(predictors)+1)+5+1)
#trace=F: reduce the amount of printed output
#maxit: number of iterations to find parameter estimates
#MaxNWts: number of parameters
#this creates a single model with 5 hidden units assuming that the data in predictors have been
#standardized to be on the same scale
nnetavg=avNNet(predictors, outcome, size=5, decay=0.01, repeats = 5, linout = TRUE,
               trace = FALSE, maxit=500, MaxNWts = 5*(ncol(predictors)+1)+5+1)
predict(nnetfit, newdata)
predict(nnetavg, newdata)
toohigh=findCorrelation(cor(solTrainXtrans), cutoff = 0.75)
trainxnnet=solTrainXtrans[,-toohigh]
testxnnet=solTestXtrans[,-toohigh]
#define the candidate models to test
nnetgrid=expand.grid(.decay=c(0, 0.01, 0.1), .size=c(1:10), .bag=FALSE)
set.seed(100)
nnettune=train(solTrainXtrans, solTrainY, method="avNNet", tuneGrid = nnetgrid, trControl = ctr1,
               preProcess = c("center", "scale"), linout = TRUE, trace = FALSE,
               MaxNWts = 10*(ncol(predictors)+1)+10+1, maxit=500)
#2. multivariate adaptive regression splines
marsfit=earth(solTrainXtrans, solTrainY)
marsfit
summary(marsfit)
marsgrid=expand.grid(.degree=1:2, .nprune=2:38)
set.seed(100)
marstuned=train(solTrainXtrans, solTrainY, method="earth", tuneGrid = marsgrid,
                trControl = trainControl(method = "cv"))
marstuned
marspredict=predict(marstuned, solTestXtrans)
head(marspredict)
#estimate the importance of each predictor in the MARS model
varImp(marstuned)
#3. Support Vector Machines
#in kernlab package, the ksvm function is available for regression models and a large number of kernel functions
#the radial basis function is the default kernel function
library(kernlab)
#if appropriate values o the cost and kernel parameters are known, the model can be fit as:
svmfit=ksvm(x=solTrainXtrans, y=solTrainY, kernel="polydot", kpar="automatic", C=1, epsilon=0.1)
#if the values are unknown, they can be estimated through resampling.
#in train, "svmRadial", "svmLinear" or "svmPoly" fit different kernels
svmrtuned=train(solTrainXtrans, solTrainY, method = "svmRadial", preProcess = c("center", "scale"),
                tuneLength = 14, trControl = trainControl(method = "cv"))
svmrtuned
svmrtuned$finalModel
#4. k-nearest neighbors
knndescr=solTrainXtrans[, -nearZeroVar(solTrainXtrans)]
set.seed(100)
knntune=train(knndescr, solTrainY, method = "knn", preProcess = c("center", "scale"),
              tuneGrid = data.frame(.k=1:20), trControl = trainControl(method = "cv"))
knntune

Tomorrow, I will do exercises on Chapter 7.

find the code

Today, I finally found the code of how to extract the remaining data from a matrix without the sample submatrix.

#biological predictors can be used to assess the quality of the raw mateiral before processing.
#manufacturing process predictors can be changed in the manufacturing process.
library(AppliedPredictiveModeling)
data(ChemicalManufacturingProcess)
ChemicalManufacturingProcess
#12 biological predictors, 45 process predictors, 176 manufacturing runs
head(ChemicalManufacturingProcess)
set.seed(1)
trainrows=createDataPartition(ChemicalManufacturingProcess[ ,1], p=0.8, list = FALSE)
trainrows
trainpredictors=ChemicalManufacturingProcess[trainrows, ]
dim(trainpredictors)
testpredictors=ChemicalManufacturingProcess[-trainrows, ]
dim(testpredictors)

The function of it is to remove the sample submatrix from the main matrix and use the remaining data to build another matrix. So we can obtain both training data matrix and testing data matrix from the data matrix that we have.

Next week, I will continue to read Chapter 7.

Chapter 7 (7.1 7.2)

Today, I read Chapter 7 (7.1 7.2).

Summary

As the regulation value increases, the fitted model becomes more smooth and less likely to over-fit the training set. Reasonable values of  range between 0 and 0.1. Since the regression coefficients are being summed, they should be on the scale; hence the predictors should be centered and scaled prior to modeling.

There are many other kinds such as models where there are more than one layer of hidden units (i.e., there is a layer of hidden units that models the other hidden units). Also, other model architectures have loops going both directions between layers.

A model similar to neural networks is self-organizing maps. This model can be used as an unsupervised, exploratory technique or in a supervised fashion for prediction.

The resulting parameter estimates are hardly to be the globally optimal estimates. As an alternative, several models can be created using different starting values and averaging the results of these models to produce a more stable prediction.

Multivariate Adaptive Regression Splines (MARS)

Once the full set of features has been created, the algorithm sequentially removes individual features that do not contribute significantly to the model equation.

GCV: generalized cross-validation

There are two tuning parameters associated with the MARS model: the degree of the features that are added to the model and the number of retained terms. The latter parameter can be automatically determined using the default pruning procedure (using GCV), set by the user or determined using an external resampling technique.

Since the GCV estimate does not reflect the uncertainty from feature election, it suffers from selection bias.

Two advantages of MARS:

1. The model automatically conducts feature selection; 2. interpretability

Tomorrow, I will continue to read Chapter 7.

Chapter 7

Today, I finished exercises of Chapter 6 and read Chapter 7.

exercise 6.3
#biological predictors can be used to assess the quality of the raw mateiral before processing.
#manufacturing process predictors can be changed in the manufacturing process.
library(AppliedPredictiveModeling)
data(ChemicalManufacturingProcess)
ChemicalManufacturingProcess
#12 biological predictors, 45 process predictors, 176 manufacturing runs
head(ChemicalManufacturingProcess)
set.seed(1)
trainrows=createDataPartition(ChemicalManufacturingProcess[ ,1], p=0.8, list = FALSE)
trainrows
trainpredictors=ChemicalManufacturingProcess[trainrows, ]
dim(trainpredictors)
testpredictors=ChemicalManufacturingProcess[-trainrows, ]
dim(testpredictors)

Summary of Chapter 7

Nonlinear Regression Models

Like PLS, the outcome is modeled by an intermediary set of unobserved variables (hidden variables or hidden units). These hidden units are linear combinations of the original predictors, but unlike PLS models, they are not estimated in a hierarchical fashion.

The linear combination is typically transformed by a nonlinear function g, such as the logistic (i.e., sigmoidal) function:

The total number of parameters is H(P+1)+H+1.

Back-propagation algorithm is a highly efficient methodology that works with derivatives to find the optimal parameters. However, it is common that a solution to this equation is not a global solution.

Weight decay (moderate over-fitting):

An alternative version of the sum of the squared errors:

Tomorrow, I will continue to read Chapter 7.

Exercise 6 (6.1 6.2)

Today, I did exercises of Chapter 6 (6.1 6.2).

6.1
#the theory of IR spectroscopy holds that unique molecular structures absorb IR frequencies differently.
library(caret)
data(tecator)
absorp
#absorbance values 215*100
endpoints
#percent of moisture, fat, and protein 215*3
#first way
corthresh=0.99
toohigh=findCorrelation(absorp, corthresh)
corrpred=names(absorp)[toohigh]
absorpeffective=absorp[, -toohigh]
absorpeffective
#second way
correlations=cor(absorp)
dim(correlations)
library(corrplot)
corrplot(correlations, order="hclust")
highcorr=findCorrelation(correlations, cutoff=0.8)
length(highcorr)
absorpeffective=absorp[, -highcorr]
absorpeffective

6.2

library(AppliedPredictiveModeling)

data(permeability)

library(caret)

lowfrequencies=nearZeroVar(fingerprints, freqCut = 20, uniqueCut = 10)

fingerfiltered=fingerprints[, -lowfrequencies]

fingerfiltered

ncol(fingerfiltered)

colnames(fingerfiltered)

#binary predictors decreases from 1107 to 413

trainfinger=fingerfiltered[sample(1:165, 132, replace=FALSE), ]

dim(trainfinger)

trainfinger

#X1=fingerfiltered[ ,1]

#X1

#trainX1=sample(X1, 132, replace = FALSE)

#trainX1

#trainfinger=fingerfiltered[trainX1, ]

#trainfinger

a=c(121, 149, 114, 40, 104, 45, 153, 26, 66, 162, 15, 128, 81, 102, 62, 127, 110, 52, 140, 95, 6, 86, 60, 11, 75,

    135, 99, 77, 34, 106, 89, 112, 87, 64, 65, 50, 59, 105, 118, 19, 94, 165, 120, 43, 48, 115, 13, 111, 41, 151, 

    116, 82, 46, 18, 38, 69, 7, 92, 23, 58, 155, 137, 132, 27, 131, 49, 93, 85, 36, 31, 79, 122, 30, 9, 145, 63, 

    143, 80, 152, 159, 134, 67, 126, 3, 70, 55, 141, 53, 101, 28, 139, 125, 109, 14, 42, 78, 147, 158, 54, 4, 117, 

    107, 150, 57, 119, 103, 156, 160, 146, 47, 29, 76, 164, 83, 73, 154, 72, 37, 124, 51, 24, 133, 22, 17, 98, 1, 

    97, 157, 148, 123, 21, 130)

testfinger=fingerfiltered[-a, ]

testfinger

(I do not know how to delete a random submatrix from a matrix and obtain the remaining one as the other matrix, I spent hours searching online but I cannot find it. I will try to solve it because it is important to my research.)

Tomorrow, I will try to figure out the problem and go on doing exercises of Chapter 6.