https://github.com/davidADSP/GDL_code/blob/master/models/VAE.py

San Diego State University

One of the most fundamental architectures for generative modeling. It was proposed in 2013 by Diederik P. Kingma and Max Welling. 

Variational AutoEncoders

Autoencoders consist of the two major parts, namely:

1.**encoder**- network compressing high-dimensional input data into lower one.
2.**decoder** - network decompressing representational vector to the original domain. 

Autoencoders can preserve as much information of the original data as possible.
The decoders can be designed to present various nice properties.
Here is the simple diagram for autoencoder:


Autoencoders

Using Variational AutoEncoders we can reconstruct images, furthermore, we can generate new images and add or subtract certain features from images.

VAE can be used to generate sequences by using a recurrent encoder and decoder, which may be able to capture more abstract level representations of data compared to other vanilla RNN models. It can also be combined with attention mechanism to be a deep recurrent attention writer model (DRAW).

VAE can also maximize importance-weighted autoencoder objectives.

Variational AutoEncoders Applications

In ordinary AutoEncoder we use *RMSE* . In case of Variational AutoEncoder we have an additional component: **Kullback-Leibler (KL) divergence**  

Variational AutoEncoder Loss Function

- VAE framework has a straightforward model architecture, it does not require extra efforts in model design compared to other frameworks, such as Boltzmann Machines.
- VAE does a better job at increasing a bound on the log-likelihood of the probabilistic model, compared to other heuristic models such as MP-DBM.
- VAE can be applied to a wide range of probabilistic models. 
- VAE is a good manifold learning algorithm as it forces the encoder to learn in different dimensions when encoder and decoder work simultaneously.

Key Advantages of Variational Autoencoders Framework

- VAE is designed to learn only one problem, that is given input data x, the reduced dimensionality Z. its mean field fixed point equation did not specify on how to share parameters for different problems.

Limitation(s) of Variational Autoencoder Framework


In a standard **autoencoder**, each input (such as an image) is mapped to a single point in the latent space. In contrast, in a **variational autoencoder** (VAE), each input is mapped to a multivariate normal distribution around a mean point in that space. The following diagram illustrates this difference:

Learn Before

Related