University of Michigan - Ann Arbor

A Basic Supervised Statistical Learning Workflow

In this step, we choose:
- A feature representation of the dataset, e.g., image pixels
- Type of classifier to use, e.g., K-nearest neighbor classifier

Classification Problem Representation

In this step, we decide on the criterion to distinguish good vs. bad classifiers. For example, the percentage of correct predictions on test set.
The learned models will be evaluated on evaluation functions (also called objective functions or scoring functions) in order to distinguish between successful and unsuccessful models. The desired, external evaluation function might be different from the one the algorithm uses.

Classification Problem Evaluation

In this step, we search for the most optimal values for (hyper)parameters that give the best classification model for the chosen evaluation criterion. For example, we can try a range of values for the "k" parameter in k-nearest neighbor classifier.

Classification Problem Optimization

A supervised learning system is expected to perform well only when a chain of four assumptions holds, each verified on a different dataset:
1. **Low bias**: the model fits the training set well on the cost function (e.g., comparable to human-level performance).
2. **Low variance**: the model fits the dev (validation) set well on the cost function.
3. The model fits the test set well on the cost function.
4. The model performs well in the real world (e.g., happy users).

Because the checks are sequential, a failure at a later step points to the stage that needs attention (e.g., high dev-set error indicates a variance problem).

Chain of Assumptions in Supervised Statistical Learning

Domingos, P. (2012). A few useful things to know about machine learning. Communications of the ACM, 55(10), 78-87. doi:10.1145/2347736.2347755

A few useful things to know about machine learning

•	Data leakage

•	Modeling non-linear data with linear models 

•	Forgetting to regularize data if needed

•	Substandard data cleaning

•	Failing to understand the underlying logic behind certain packages

Learn Before

Related