Concept of LoRA or Low Rank Adapation in LLMs

LoRA is an approach to optimizing LLMs by reducing the "size" of the matrix of trainable parameters, as measured by "rank" of the matrix i.e. the number of linearly independent rows or columns.