Statistical Similarity

KL divergence is a way to measure "closeness" of two probability distributions. It is used in the literature on model cloning/model distillation.

Another name for it is "relative entropy".

Its full form is Kullback-Leibler divergence. 

It is denoted D[KL]( P || Q ). P is the true probability distribution and Q is the approximating probability distribution.

Mathematically:

D[