About Me

Low-Rank Adaptation (LoRA) efficiently fine-tunes large pre-trained language models through low-rank weight updates, significantly reducing memory usage compared to full fine-tuning. However, the problem of how to optimally allocate low-rank parameters across model layers remains challenging. This paper presents an extended theoretical framework that unifies classical matrix approximation perspectives with data- and curvature-aware approaches, developing both offline and online rank-allocation a ...