The Random Forest is a model made up of many decision trees. It trains each one on a slightly different set of the observations, splitting nodes in each tree considering a limited number of the features. The final predictions of the random forest are made by averaging the predictions of each individual tree. In random forest algorithm, we do not need to do pruning for each "weak" decision tree. 

Advantage of Random Forest:
- Less overfitting
- Parallel implementation

Random Forest

Support Vector Machines are intended for binary classification (cases where there are two classes). The main goal is to find a hyperplane that best divides a dataset into two classes. This is quantified by achieving the maximum distance between data points of both classes separated by the hyperplane. The further away the data points are from the hyperplane, the more confidently we are able to classify future data points into their respective class.

Support Vector Machines

Boosting is another approach that can be used for both classification and regression. Boosting improves predictions of a decision tree by learning slowly. Unlike bagging, boosting works sequentially, where each tree grown uses the information from those created before it, and works on modified versions of the original data set. Boosting combines a large number of trees ($\hat{f}^{1}, ...,  \hat{f}^{B}$). Decision trees are fitted based on the residuals of the model rather than the response (Y), then update the residuals by adding it to the fitted function. This means that each model created is slowly reducing the error of the previous ones. This causes an improvement of $\hat{f}$, particularly in places where it does not do as well as others. Decision trees using boosting tend to be very small.


Boosting

A decision tree divides the prediction space into a series of simple regions according to the stratifying criteria. Decision trees can be applied to both regression and classification problems and can thus be of two types.

Decision Tree

Tree based methods aims to stratify or segment the predictor space (two predictors expressed in x y plane) into several simple regions. We can make a prediction for an observation by taking the mean or mode of the training observations in the region to which the observation belongs.

Tree based methods can be used both for 
-	Regression 
-	Classification


        


Tree-based Methods 

An Artificial Neural Network (ANN) is a computational model inspired by the structure of biological neural networks in the brain, made up of interconnected neuron-like units organized in layers. Rather than following task-specific rules, an ANN learns to recognize patterns in data from examples, adjusting the connections between its units during training.

Artificial Neural Networks

Reference to Artificial Neural Networks

A method of supervised classification that takes many features, assumed to be independent, and uses Bayes' Theorem to determine given those independent variables how likely the likelihood of another event occurring.

It's called "Naive" because it assumes that features are conditionally independent, given the class, i.e., for all instances of a given class, the features have little or no correlation with each other.

(Naive) Bayes Classifier

Generalized Additive Models (GAMs), similar to multiple linear regression models, attempt to predict a response value ($$Y$$) using many predictor variables ($$X_1, \dots, X_p$$). GAMs maintain their additivity by applying separate non-linear functions to each predictor variable ($$f_j(X_j)$$) and adding them together. They can be applied to both quantitative and qualitative responses.

Generalized Additive Models (GAMs)

- Generalized Additive Models
- Trees
- Random Forests
- Boosting
- Support Vector Machines
- Naive Bayes Classifier

University of Michigan - Ann Arbor

Classifiers are techniques used to predict qualitative variables.
- Methods that are not very computer intensive (widely-used)
- Methods that are much more computer intensive
- Decision Tree that can be used on regression and classification problems

Methods of Classification

- GAM
- Boosting
- Support vector machines

Modern SL Methods (examples)

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

- Logistic Regression
- Linear Discriminat Analysis
- K-nearest Neighbors

Most Common Classifiers

Generalized additive models (GAMs) provide a general framework for
generalized additive model extending a standard linear model by allowing non-linear functions of each of the variables, while maintaining additivity. 

Generalized Additive Models

Boosting algorithms are methods to help transform weak learners to stronger learners. There are 3 common boosting algorithms used in data science:

1. AdaBoosting (Adaptive Boosting)
2. GradientBoosting
3. XGBoost (Extreme Gradient Boosting)

Learn Before

Related

Learn After