Generative Models - GANs/VAEs and Diffusion Models - O My

 GANs are obviously generative adversarial networks. 

• Developed by Ian Goodfellow and colleagues in June 2014
• In this construct, two neural networks compete with each other in the form of a zero-sum game, where one agent's again is the other agent's loss
• The concept is that competition forces both to get better
• The game is to provide high realism output
• They are used where you need hi-fidelity synthetic data, realistic imagery
• They are not the dominant model for consumer generative AI
• The theory behind GANs is an interesting application of probability spaces

Diffusion models.

• Class of latent variable generative models
• Based on diffusion processes in applied probability
• The goal of the model is the learn the diffusion process that underpin an image (strange concept in itself, and one that puts this technique in the realm of latent variable generative models)
• Two components - the forward diffusion process and the reverse sampling process
• Simple example of a diffusion process is ink dropped in water, droplets diffuse through the water
• One example application is denoising images (where image is blurred with Gaussian noise)
• Stable Diffusion and DALL-E are diffusion based image generators

VAEs are variational auto-encoders

• A variational auto-encoder (VAE) is an artificial neural network introduced by Kingma and Welling in 2013
• It is part of the families of probabilistic graphical models and variational Bayesian methods

** classes of model **

Latent variable generative models.

• Statistical model that relates a set of observable variables (also called manifest variables, indicators) to a set of latent variables
• Latent variables are those that can be observed /inferred only via a mathematical model
• They may correspond to aspects of physical reality
• Earliest reference: Francis Bacon, Novum Organum
• It may reduce the dimensionality of the data

Auto-encoder.

• Form of artificial neural network.
• Used to learn efficient codings of unlabeled data (unsupervised learning)
• An autoencoder learns two things:
• Encoding function - transforms the input data 
• Decoding function - that recreates the input data from the encoding/encoded representation
• Autoencoder learns an efficient representation (encoding) for a set of data, typically for dimensionality reduction, to generate lower dimensional embeddings for subsequent use by machine learning algorithms

Variations of auto-encoders that embody useful properties.

• Example: regularized auto-encoder (aka sparse, denoising and contractive autoencoders)
• Example: variational auto-encoder (used for generative applications)

"A continued process which... escapes the observation of the senses" - Francis Bacon