LoRA in Real Workflows

LoRA, or low-rank adaptation, is a fine-tuning technique for LLMs (one of many disparate techniques). 

The idea is to inject low rank matrices into large pre training models.

Recall that the rank of a matrix A is the dimension of the vector space spanned by its columns. This in turn corresponds to the number of linearly independent columns of A.

So LoRA is essentially a dimensionality reduction of the column space of parameters to ease off compute.