One of the most fundamental architectures for generative modeling. It was proposed in 2013 by Diederik P. Kingma and Max Welling. 

Variational AutoEncoders

This generative model scheme is set apart from MCMC sampling and ancestral sampling because it is based on nonequilibrium thermodynamics. This approach assumes that the probabilistic distribution we will sample from has some structure. This structure is gradually broken down by some diffusion that increases the entropy in the distribution thus making it unstructured. 


To create the generative model, we then train a model that reverses the process and gradually returns structure to our distribution. Similarly to MCMC, we apply many iterations to produce our sample; however, in this case our model is iterating in order to transform the entropy-filled distribution to our target distribution.   


Diffusion Inversion

The ABC approach rejects or modifies samples to make the moments of the selected samples match the desired distribution. Unlike moment matching, this approach does not train the mode to automatically produce samples with the correct moments but rather it modifies the sample directly. A simple explanation is given below: 

Approximate Bayesian computation (ABC) framework

1. Variational AutoEncoders (_VAE_)
2. Generative Adversarial Networks (_GAN_)
3. Diffusion Inversion
4. Approximate Bayesian computation (_ABC_) framework

Nova Southeastern University

San Diego State University

**Generative Models**  As might be clear from their name , generative models are able to generate new contents by learning features of certain datasets. As for more formal definition, generative models specify in terms of a probabilistic model describing how it thinks the dataset was produced.  Using sampling from this probability distribution, we can create new data in the style of the old.
 
The joint probability distribution P(X,Y) is learned from the data, and then divided by P(X) to obtain the conditional probability distribution P(Y|X), which is used as a predictive model. It is called a generative model because the model represents the relationship between a given input X and an output Y. Common generative models are: Naive Bayes, Hidden Markov Model, Gaussian Mixture Model, Document Topic Generation Model (LDA), and restricted Boltzmann machine.

Generative Models

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

If the model is based on fixed calculations we will have the same results every time(i.e. taking average value of each pixel), thus that's not generative. It is inevitable to use random element influencing each samples that are produced by our model. 
(**quick note** this comparison can be  similar to *linear* vs *non-linear* activation functions in deep learning)

*Probabilistic* rather than *Deterministic*

Most problems that you might have faced in the machine learning are discriminative in their nature. Unlike generative modeling here each sample has a **label** .  You can consider discriminative modeling as supervised learning.

The discriminative model learns the decision function f(X) directly from the data, or the conditional probability distribution P(Y|X) as the prediction model. The discriminative model is concerned with what output Y should be predicted for a given input X. Common discriminative models are: K-nearest neighbor, perceptron, linear regression, linear discriminant analysis (LDA), LR, SVM, decision tree, neural network, boosting, conditional random field, maximum entropy model.

In order to understand difference between *discriminative* and *generative* modeling you can use these definitions:
 

Discriminative Modeling

1. Complete understanding of how the data was generated, not merely categorizing objects.

2. Seems highly promising in the Reinforcement Learning 

3. It is mightier to acquire machine with  the form of human intelligence. 

Why *Generative Modeling* ? 

1. _Sample Space_ is set of all values that observation x can take.
2. _PDF_ (Probability Density Function) is the function mapping point from sample space to the number between 0 and 1. Area under PDF is equal to 1.
3. _Parametric Models_ are density functions having finite number of parameters.
4. _Likelihood _ is defined as $L(\theta | x) = p_{\theta}(x)$
5. _Maximum Likelihood Estimation_ is defined as $\theta = argmax L(\theta | X)$.
It is used to measure the set of parameters that are more apt to explain observed data X.

Quick Recap For Some Probability Concepts

Representational learning is a class of machine learning that focuses on automatically discovering the most appropriate way to represent data. By learning these representations directly from the data, it eliminates the need for manual feature engineering and allows models to effectively process and understand complex inputs.

Representational Learning

https://www.amazon.com/Generative-Deep-Learning-Teaching-Machines/dp/1492041947

David Foster's Generative Deep Learning

Deep Belief Networks (DBNs) were one of the first nonconvolutional models to successfully admit training of deep architectures. They are generative models with several layers of latent or hidden variables in which the latent variables are typically binary, while the visible units may be binary or real. Every unit in each layer is connected to every unit in each neighboring layer.

Deep Belief Networks (DBNs)

Practitioners usually evaluate generative models by having experimental subjects visually assess the samples. This can help to notice under and overfitting, but struggles because bad models may still look solid visually. In this case, databases like MNIST are used to continue to check.

Evaluating Generative Models

Generative Adversarial Networks are a type of generative modeling approach. In this process, the generator network competes with the discriminator network, which attempts to distinguish between samples drawn from either the training data or the generator. This allows the discriminator to produce a probability that a particular training example is real rather than a fake sample drawn from the model. 

Generative Adversarial Networks

A generator network using a convolutional structure is often useful for generating images. We utilize the "transpose" of the convolution operator, often yielding more realistic images using fewer parameters.

Convolutional Generative Networks

A **Generative Stochastic Network**, or **GSN**, is a generative model with latent or hidden variables $$\bold h$$ in the Markov chain, in addition to the ordinary visible random variables $$\bold x$$.

Generative Stochastic Networks (GSNs)

Here is the example to help make the idea of a generative model comprehensible for you. Suppose you have dog images and you want to generate a dog image that looks real but that dog actually doesn't exist. This can be achieved by learning the characteristics of dog images. This is a typical problem which we can solve using generative modeling.

Here is a visual clue for you:

Generative Model Example

•	The standard procedure for drawing samples from a normal distribution with mean µ and covariance Σ  is to feed samples z from a normal distribution with zero mean and identity covariance into a very simple generator. 
•	Pseudorandom number generators can also use nonlinear transformations of simple distributions. In this case, we can use the inverse transform sampling techniques. If we can specify p(x), integrate over x, and invert the resulting function. In this case, we don’t need to use machine learning methods.


How to generate samples from not complicated distributions using generator networks?

• To generate samples from more complicated distributions that are diﬃcult to specify directly, diﬃcult to integrate over, or whose resulting integrals are diﬃcult to invert, we use a feedforward network to represent a parametric family of nonlinear functions g, and use training data to infer the parameters selecting the desired function

Generate samples from complicated distributions

• When emitting the parameters of a conditional distribution over x, it can generate discrete data as well as continuous data. However, these data could be algebraically manipulated by humans, which is a weakness.

• On the other hand, when the generator net provides samples directly, it can generate only continuous data, and are not forced to use conditional distribution. The problem with direct sampling is that the model could no longer be trained using back-propagation. 


Emitting the parameters of a conditional distribution versus directly emitting samples

Some generative models allow the probability density function to be evaluated explicitly; others do not. Some are structured probabilistic models which can be described in terms of graphs and factors; others cannot, though they still represent probability distributions.

Variations of generative models

Which of the following models are machine learning generative models?

Generative models

The learning process of generative modeling requires optimizing intractable criteria. In the context of differentiable generator nets, the criteria are intractable because the data does not specify both the inputs $$z$$ and the outputs $$x$$ of the generator net. However, in the case of supervised learning, both the inputs $$x$$ and the outputs $$y$$ are given, and the optimization procedure needs only to learn how to produce the specified mapping.

Learn Before

Related

Learn After