Pros:
- With GAMS, one can fit non-linear functions to each predictor to automatically model non-linear relationships that standard linear regression will miss
- The non-linear fits *can* make more accurate predictions for the response Y
- The additive nature of GAMs means we can still examine the effect of each predictor variable on Y while holding other variables fixed.
- Degrees of freedom can summarize the smoothness of the GAM

Cons:
- The main limitation is that GAMs are restricted to be additive, meaning important interactions *can be missed*. However, as with regular linear regression, we can manually add interactions by including additional predictors.













Pros and Cons of GAMS

For Generalized Additive Models (GAMs) in regression problems, to depict a non-linear relationship between predictors and their response, we start with the multiple linear regression model $$y_i = \beta_0 + \beta_1x_{i1} + \beta_2x_{i2} + \dots + \beta_px_{ip} + \epsilon_i$$. We then replace each linear component ($$\beta_jx_{ij}$$) with a non-linear function $$f_j(x_{ij})$$ so that the model becomes $$y_i = \beta_0 + f_1(x_{i1}) + f_2(x_{i2}) + \dots + f_p(x_{ip}) + \epsilon_i$$. A separate non-linear function $$f_j$$ is applied to every predictor variable, and these are then added together. This is why it is considered an additive model.

GAMs in Regression Problems

Similar to their use in regression, Generalized Additive Models (GAMs) can be applied to classification problems with qualitative responses. Assuming a binary response $$Y$$ that equals 0 or 1, we can extend the standard logistic regression model, $$\log\left(\frac{p(X)}{1-p(X)}\right) = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \dots + \beta_p X_p$$, by replacing each linear component ($$\beta_j X_j$$) with a non-linear function ($$f_j(X_j)$$). The resulting GAM for classification is: $$\log\left(\frac{p(X)}{1-p(X)}\right) = \beta_0 + f_1(X_1) + f_2(X_2) + \dots + f_p(X_p)$$.

GAMs for Classification Problems

https://codeburst.io/pygam-getting-started-with-generalized-additive-models-in-python-457df5b4705f



GAM implementation in Python

For the `Wage` data, an additive model expressed with explicit spline basis functions can be fitted using `lm()`:

```R
gam1 <- lm(wage ~ ns(age, 5) + education, data = Wage)
```

To specify smooth terms with `s()`, use `gam()` from the `gam` library:

```R
library(gam)
gam2 <- gam(wage ~ s(year, 4) + s(age, 5) + education, data = Wage)
```

Fitting generalized additive models in R

Generalized Additive Models (GAMs), similar to multiple linear regression models, attempt to predict a response value ($$Y$$) using many predictor variables ($$X_1, \dots, X_p$$). GAMs maintain their additivity by applying separate non-linear functions to each predictor variable ($$f_j(X_j)$$) and adding them together. They can be applied to both quantitative and qualitative responses.

University of Michigan - Ann Arbor

Google

- Polynomial Regression
- Step Functions
- Regression splines 
- Smoothing splines 
- Local regression
- Generalized additive models

Nonlinear regression methods

- Generalized Additive Models
- Trees
- Random Forests
- Boosting
- Support Vector Machines
- Naive Bayes Classifier

Computer Intensive Classifiers

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Local regression works by choosing a target point ($$x_0$$) and its nearby training observations in order to find a fit. Points are then assigned weights (given by $$K_{i0} = K(x_i, x_0)$$) based on their distance from $$x_0$$; points closest to $$x_0$$ are assigned the highest weight, and as points get farther from $$x_0$$, they are assigned a lower weight. The point farthest from $$x_0$$ will be assigned a weight of 0. A weighted least squares regression is then performed by minimizing $$\sum_{i=1}^{n} K_{i0} (y_i - \beta_0- \beta_1x_i)^{2}$$. The new fitted value at $$x_0$$ is then found using the new regression model.

Local Regression

In order to fit a smooth curve of a set of data, we need to find a function g(x) to fit the observed data well. We have to make g to be able to keep $RSS = \sum_{i=1}^{n} (y_i – g(x_i))^2$ as small as possible and the curve as smooth as possible. 

Minimize function g: Loss & Penalty; $\lambda$ is tuning parameter
$\sum_{i=1}^{n} (y_i – g(x_i))^2 + \lambda \int g^n(t)^2 \, dt$


Smoothing Splines

A step-function model replaces one global linear fit with several flat pieces. First, split the predictor range into intervals, or bins. Then estimate a separate constant response level within each bin.

Because the fitted value changes only when the predictor crosses a bin boundary, the model is easy to interpret: each interval has its own average outcome. This makes a continuous variable behave like an ordered categorical variable, which can be helpful when the response changes in distinct ranges rather than smoothly.

Step Functions

Polynomial curve fitting is a linear regression problem where the features are given by the polynomial powers of the original input. For a single feature $$x$$ and a corresponding real-valued label $$y$$, the goal is to estimate a polynomial of degree $$d$$ using the formula $$\hat{y}= \sum_{i=0}^d x^i w_i$$. Here, $$w_i$$ represents the model's weights and $$w_0$$ acts as the bias (since $$x^0 = 1$$ for all $$x$$). Because it is a linear regression problem, the squared error is typically used as the loss function.

Polynomial Curve Fitting

Regression splines model a nonlinear relationship by splitting the range of a predictor variable X into K intervals and fitting a separate polynomial on each interval. The pieces are estimated so that the final curve joins smoothly at the knot locations. For example, if X runs from 0 to 100 and knots are placed at 25, 50, and 75, one polynomial is fit on 0-25, another on 25-50, and so on, with the pieces connected at each knot.

How Regression Splines Work

The basis function approach to regression involves using a family of functions or transformations that can be applied to a variable $$X$$: $$b_1(X), b_2(X), \cdots, b_K(X)$$. Instead of fitting a standard linear model in $$X$$, we fit the model: $$y_i = \beta_0 + \beta_1b_1(x_i) + \beta_2b_2(x_i) + \cdots + \beta_Kb_K(x_i) + \epsilon_i$$. Note that the basis functions $$b_1(\cdot), b_2(\cdot), \cdots, b_K(\cdot)$$ are fixed and known, meaning they are chosen ahead of time.

Basis Functions for Regression

The Random Forest is a model made up of many decision trees. It trains each one on a slightly different set of the observations, splitting nodes in each tree considering a limited number of the features. The final predictions of the random forest are made by averaging the predictions of each individual tree. In random forest algorithm, we do not need to do pruning for each "weak" decision tree. 

Advantage of Random Forest:
- Less overfitting
- Parallel implementation

Random Forest

Support Vector Machines are intended for binary classification (cases where there are two classes). The main goal is to find a hyperplane that best divides a dataset into two classes. This is quantified by achieving the maximum distance between data points of both classes separated by the hyperplane. The further away the data points are from the hyperplane, the more confidently we are able to classify future data points into their respective class.

Support Vector Machines

Boosting is another approach that can be used for both classification and regression. Boosting improves predictions of a decision tree by learning slowly. Unlike bagging, boosting works sequentially, where each tree grown uses the information from those created before it, and works on modified versions of the original data set. Boosting combines a large number of trees ($\hat{f}^{1}, ...,  \hat{f}^{B}$). Decision trees are fitted based on the residuals of the model rather than the response (Y), then update the residuals by adding it to the fitted function. This means that each model created is slowly reducing the error of the previous ones. This causes an improvement of $\hat{f}$, particularly in places where it does not do as well as others. Decision trees using boosting tend to be very small.


Boosting

A decision tree divides the prediction space into a series of simple regions according to the stratifying criteria. Decision trees can be applied to both regression and classification problems and can thus be of two types.

Decision Tree

Tree based methods aims to stratify or segment the predictor space (two predictors expressed in x y plane) into several simple regions. We can make a prediction for an observation by taking the mean or mode of the training observations in the region to which the observation belongs.

Tree based methods can be used both for 
-	Regression 
-	Classification


        


Tree-based Methods 

An Artificial Neural Network (ANN) is a computational model inspired by the structure of biological neural networks in the brain, made up of interconnected neuron-like units organized in layers. Rather than following task-specific rules, an ANN learns to recognize patterns in data from examples, adjusting the connections between its units during training.

Artificial Neural Networks

Reference to Artificial Neural Networks

A method of supervised classification that takes many features, assumed to be independent, and uses Bayes' Theorem to determine given those independent variables how likely the likelihood of another event occurring.

It's called "Naive" because it assumes that features are conditionally independent, given the class, i.e., for all instances of a given class, the features have little or no correlation with each other.

Learn Before

Related

Learn After