Recent work like Git Re-Basin suggests that models trained independently from scratch can often be aligned through permutations, indicating a shared underlying structure. However, this phenomenon is primarily observed in wider networks. I'm interested in understanding how model width influences when such alignment is possible, and more broadly, how to improve model matching in modern architectures like transformers, especially at scale.

Large collections of open-source model checkpoints offer a rich resource for understanding how models adapt across tasks. Methods like LoRA suggest that these adaptations can often be captured through low-dimensional updates rather than full retraining. However, from a weight-space perspective, such parameterizations are not unique—symmetries imply that many equivalent representations of these updates may exist across the model zoo, making weight-space learning challenging. I’m currently interested in developing weight-space learning approaches for LoRA that leverage this shared structure for efficient LLM adaptation.