04 · CAPACITY
VerifiedRank, Alpha, and Target Modules
Separate capacity, scale, and intervention location; change one variable at a time.
- r = capacity
- alpha = scale
- α/r standard, α/√r rsLoRA
- Change one variable at a time
KNOW THESE FIRST
The budget, window, and …Rank r determines adapter capacity and the r × (d_in + d_out) parameter budget. Alpha controls alpha/r in standard LoRA and alpha/√r in rsLoRA.
Target modules decide which real attention and MLP matrices receive adapters. Names are architecture-specific and must be verified on the model.
First thought
The same alpha/r means the same adapter capacity.
Correction
Effective scale may match while a higher rank still carries more parameters and capacity.
Decision rule
Start at r=8/alpha=8; if capacity is insufficient, hold everything else fixed and test r=16/alpha=16.
CONCEPT DEPTH
Read the same concept at three depths. Pick a level, switch instantly. Recommended starting point for university students: UNDERGRAD.
Rank r is the inner dimension of A ∈ R^(r×d_in) and B ∈ R^(d_out×r). Parameter count scales as r × (d_in + d_out). Alpha is the scale factor: α/r in standard LoRA, α/√r in rsLoRA. Even if effective scale is held constant, higher rank carries more capacity and increases overfitting risk.