Capacity of a LoRA Adapter (Part 1)

Introduction

Finetuning is a commonly applied paradigm for adapting language models (LLMs) to downstream tasks via specifically designed data. In 2021, Hu et al. observes that this update is approximately low rank, making a full update on model parameters unnecessary.

Specifically, for a weight matrix $W_0 \in \mathbb{R}^{d\times k}$, LoRA replaces the dense update $\Delta W$ with a low-rank factorization,

where $B \in \mathbb{R}^{d\times r}$, $A \in \mathbb{R}^{r\times k}$, and $r \ll \min(d,k)$. The scalar $\alpha$ is a fixed scale set once and not trained ($\alpha = 1$), and dividing it by rank $r$ keeps the size of the update roughly constant as the rank changes. This is a nice proposal that allows one learning rate to work well across $r$ choices.

Since only $A$ and $B$ get trained while $W_0$ stays frozen, LoRA drops the trainable count from $dk$ down to $r(d+k)$ per weight matrix. The result is a small, detachable patch (also called an “adapter”) $\{A, B\}$ pair that conserves memory in training.

A playable animation for LoRA decomposition

Even at the largest rank, the adapter $\{A, B\}$ stays a small fraction of $W_0$. This saving has popularized LoRA to be the canonical post-training method. In many deployments, it is effectively the only method. For example, a serving stack that hot-swaps thousands of task-specific adapters over one base. Whenever you are committed to an adapter, the question that decides whether finetuning will work is a capacity question: since the patch is a sliver of the model, how much can it actually hold?

In this set of experiments, I ask: how much data can a LoRA adapter store, and to what degree can its capacity be linked to the base model?

Morris et al. (2025) - one of my favorite papers this year! - provide a way to quantify such a capacity question. They separate what a model memorizes from what it generalizes by training on random facts, where by memorization can directly be inferred by the number of data bits an adapter can reproduce. Notably, here the author concludes that a full model holds about 3.6 bits of information per parameter.

DataDecide, a suite of 10,000 checkpoints we released with Ai2 last year, can be a nice bed of resources to answer these questions. Across a few model sizes, their step checkpoints, and tasks, we found:

Measuring capacity via Memorization

One way to make capacity concrete is to look at it through memorization: count the bits a model is asked to absorb during finetuning, then ask how many of them actually get observed in the weights. Memorization is less glamorous than generalization, but in a good model the two are often tightly coupled. To get even more complex, model might also learn a way to compose the facts that it holds, making memorization and skills inseparable (Morris et al). A frontier model cannot answer Terminal Bench questions without knowing the syntax of grep/find/cd, and some logical ways to combine them.

The only way to separate Memorization and Generalization is to make the bits maximally ungrokkable.

Setup: To this end, we build a pool of random (entity, relation, value) by sampling from the OLMo tokenizer. 4,000 entities x 800 attributes x 4,000 values give us a total of 3.2 million facts. Every fact holds approximately 12 bit, equating to ~4.8 MB of data. For ease of training, we write them as short sentences with a single-token answer, like so:

Each entity and value is a real word that the OLMo tokenizer encodes as a single token in its vocab, keeping the answer logit in a single position. Given this setup, the only way for the model to complete Attribute 61 of Perry is __ is to have memorized Yugoslav.

Since the value is drawn uniformly from $V = 4{,}000$ candidates, a model that has never seen it can do no better than guess, and naming the right one from scratch costs $\log_2 V \approx 12$ bits. Learning one fact is worth about 12 bits. A model that has learned the fact spends fewer bits to store it, with the savings made possible by the trained weights.

Formally, we can test exactly the unintended-memorization measure suggested in Morris et al. as: $\text{bits} = \log_2 V - \text{NLL}_2(\text{answer} \mid \text{query})$, the uniform prior over the $V$ possible values minus whatever it still gets wrong.¹

The capacity ceiling

To measure how many $bits$ a LoRA adapter can hold, we train each model and adapter rank on a growing set of $D$ facts and read off the bits it memorizes. All experiments stay in bfloat16 and seq=256.

Capacity $C$ is the maximum, over dataset size $D$, of the total bits the trained adapter memorizes:

where the sum runs over the $D$ facts in the training set. Below a critical $D$ the post-trained model rides the diagonal and stores every fact, so bits just track $D$. Past it, storage saturates and then declines as new facts overwrite old ones. That plateau is the capacity, and the two regimes meet at $D_c = C/\log_2 V$.

We visualize the capacity of each base and rank setups via their memorization curves below.

Bits memorized against training-set size for the 530M base, one line per LoRA rank, log-log.

Each curve rides the diagonal while it has room, then peels off at its own ceiling. Intuitively, smaller bases peel sooner, the 20M near ten thousand facts and the 530M past a hundred thousand. At rank 4, the 20M adapter crosses from about 104 kbit at 18k facts to roughly -39 kbit at 400k, having overwritten more than it kept. Pushed far past capacity, the smallest model even does worse than random guessing, where its stored bits turn negative. Schulman et al.’s LoRA Without Regret sees the first half of this shape on real tasks: loss curves track full fine-tuning until a rank-dependent threshold, then fall off as the adapter runs out of capacity. The overshoot into negative territory, though, is ours; in their setting a past-capacity LoRA learns less efficiently rather than regressing.

The capacity law

How can we predict the capacity of a given base and adapter pair before training? Naturally, we can turn to scaling laws. Some dimensions we discuss thus far provide valuable signals: the base parameter count $N$, number of tokens $T$ saw in pretraining, and the adapter’s trainable parameter count $p \approx 2\,r\,d_\text{model} \times (\text{layers} \times \text{matrices})$, with the adapter on every attention and MLP projection. Since DataDecide supplies matched checkpoints across abundant model sizes and pretraining steps (dolma1_7 data recipe, roughly 100 tokens per parameter), we can fit a scaling law over every (base, rank) cell.

We posit a joint power law, $C \propto N^{a}\, p^{b}\, T^{c}$, and estimate the exponents by least squares in log space,

We can visualize the performance of fitting on different variable combinations:

Our fit is done on models up to 530M, with 1B held out. The $N \cdot p$ fit uses the 25 finals (≤530M, ranks 1–16); the $p \cdot T$ and $N \cdot p \cdot T$ fits add the pretrain step checkpoints (all six sizes, ranks 1/4/8/16). With any fit involving T, we use 101 points.

Two findings stand out. First, the exponent on $p$ lands near one in every view ($0.97$ to $1.08$): capacity is essentially linear in the adapter’s trainable parameters. Second, one variable on model quality is adequate. With pretraining tokens in the fit, the exponent on $N$ turns slightly negative, so the base’s raw size adds no predictive power on top:

Dropping $T$ lets size stand in as its proxy at nearly the same quality,

In “Distillation Scaling Laws”, Busbridge et al. also treat base size and pretraining tokens as two handles on the base’s quality, where teacher size and teacher tokens reach the student only through the teacher’s loss. It seems we can use $T$ when the pretraining budget is known, but otherwise $N$ can stand in: in DataDecide, models are trained to roughly 100 tokens per model parameter, about 5x Chinchilla-optimal.

Is the base-quality term causal, or is $N$ merely correlated with it? A bigger base is also a more heavily pretrained one, so the clean test holds each base and adapter fixed and varies only how long the base was pretrained, walking DataDecide’s checkpoints from random initialization to the final model (plotted as capacity per adapter parameter, $C/p$, to put all sizes on one scale).

Capacity rises monotonically with pretraining, at every size and rank. A random base sits at the injection floor, near 0.01 bits per parameter, because it has no representations to remap; pretraining lifts it severalfold, from about 0.05 at 20M to roughly 0.2 at the mid sizes and 0.4 at 1B by the end of the sweep. Holding size and parameter count fixed while adding tokens still raises capacity, so the base-quality term in the law is causal, not just a correlation with size. The lift shows up across every rank measured (1, 4, 8, 16), not just one.

When fit on models up to 530M, the law predicts 1B’s capacity within 0.8 to 1.4 times measured (the held-out $R^2 = 0.96$). Notably, a fit on $p$ alone collapses on the held-out 1B ($R^2 = 0.45$). Adding either $N$ or $T$ as an axis restores our predictability. I have found that a fit on <=150M can work almost equally well ($R^2 = 0.92$ for predicting 1B), so a cheap budget might suffice in practice.

With the law, we can quantify precisely the capacity of an adapter rank. Because the exponent on $p$ is essentially one ($p^{1.08}$), dividing the fit through by $p$ gives the bits per LoRA parameter,

Since the exponent on $p$ is small, density is mostly a property of the base model instead of the adapter size: at 530M it sits between 0.13 and 0.20 across ranks 1 to 16, while the base moves it from about 0.05 at 20M to 0.4 at 1B. Specifically:

Thus, in our suite, a LoRA parameter holds about 0.05 to 0.4 bits per trainable parameter.

How do these densities compare to full training? Morris et al. train GPT-family models from scratch on random data, the same protocol we borrow here, and read a plateau of about 3.6 bits per parameter (also in bfloat16). Allen-Zhu and Li get at the number differently: they train on synthetic biographies and count how many facts can be probed back out, landing near 2 bits per parameter. Against either reading, a LoRA is roughly an order of magnitude less dense than the weights it patches.

Even at 1B the density sits well below the full training ceiling, but a low density is not a weakness in itself. Thinky’s blog post corroborates with the conclusion that a LoRA keeps pace with full finetuning until the finetuning set asks for more bits than the adapter can hold. In the next section, we examine more closely where the capacity boundary falls from the perspective of recall.

From capacity to recall

Our $bits$ and capacity ceiling for LoRA have closely followed Morris’s framing. In practice, we might want to also gage model’s recall ability given that it has been train on a set of $D$ facts. Here, capacity governs both storage (our “bits” defined in previous session) and recall – simply asking how many facts model can guess correct.

For a single base and rank, both stored bits and recall move with dataset size in units of capacity $D_c = C/\log_2 V$. Below $D_c$ the adapter recalls almost every fact and stored bits rise; around $D_c$ storage saturates at $C$ and recall bends over together. Storage stops rising at the same dataset size where recall starts falling, sharing capacity as the yardstick for the size of the training set.²

In the following plot, we collapse our family onto one curve by dividing each curve’s dataset by its capacity in facts, $D_c = C/\log_2 V$.

Every memorization curve, rescaled by its own capacity in facts $D_c = C/\log_2 V$. The sigmoid is fit on the ≤530M cells alone. Shading marks the regimes: store ($D < D_c$), lose ($D_c$ to $5D_c$), collapse ($> 5D_c$).

Recall stays near one before crossing $D_c$, halves when passing it, and is gone by about $5\,D_c$. The same behavior applies at almost every size and rank, with a single shared steepness $k \approx 2.3$.

By substituting in capacity, we can fit Recall as function of the base, the adapter, and the dataset directly,

Before training, we can – similar to scaling law – apply the fact set size to this equation. Below that $C(N,p)/\log_2 V$, the adapter keeps everything. Above it, it will start trading old facts for new.

Real-world post-training

The synthetic law makes rank buy capacity: $D$ structureless facts force an update spanning on the order of $D$ directions, so a bigger adapter stores more. Does that carry over to real fine-tuning? Does a task with more data, or more information-dense data, need more rank? To find out we ran a spread of real tasks and, for each, measured two things: the reduction in test loss each LoRA rank buys, and the effective rank of the update $\Delta W = BA$ the adapter actually learns. The synthetic random facts are the memorization control at one extreme.

We evaluate on the following tasks:

• random facts: the synthetic $(\text{entity}, \text{relation}, \text{value})$ triplets defined earlier, with each fact worth 12 bits and no shared structure to exploit.
• Aya: multilingual instruction following, teaching an English base model a language it has barely seen. Knowledge injection, the high-information extreme.
• PopQA: real Wikidata facts phrased as questions, split by entity popularity. Very close to our synthetic set and is fully grounded.
• Super-NaturalInstructions: a pool of genuinely different instruction task types (translation, arithmetic, NER, summarization, sentiment, and more).

Setup. Every adapter uses LoRA on all linear layers (attention and MLP), trained with each task’s own objective. In metrics, we consider:

1. Test loss, the held-out negative log-likelihood in bits/token, over the frozen base at each rank
2. Effective rank of the update $\Delta W$, which counts how many singular directions carry “real” weight. We compute it as $\exp\big(-\sum_i p_i \log p_i\big)$, the perplexity of the normalized singular values $p_i = s_i/\sum_j s_j$, and take the median over the 65 adapted matrices. A low effective rank means one shared transformation; a high one means many independent directions.

Aya: Injecting multilinguality

The first task is multilingual instruction tuning on the Aya Collection, thanks to the great effort from Cohere Labs community. Given DataDecide pretraining is primarily English we deem this to be an information-heavy task where model might memorize language the way it memorizes random facts.

Hypothesis. The capacity law helps us understand the budget before any training. Plugging the 1B base and each adapter into $C \propto N^{0.38}\,p^{1.08}$ outputs about 190 kbit at rank 1 and 3.7 Mbit at rank 16. We use test loss on the language as the cost of storing the per-language token, which lands in the range of 2 to 6 bits per token (third column of the table below). We can call this 4, which gives us a quick napkin math for predicted capacity:

If the adapter stored language the way it stores random facts, rank 1 would collapse and rank 16 would pull far ahead.

Result. Our hypothesis on the effectiveness of rank on Aya is negative. What moves the gain is how unfamiliar the language is to the base (reduction in test loss, bits/token, at 25.6k examples). Rank 1 matches rank 16 within $0.05$ bits in all eight cells, even at 170× its supposed storage budget. The gain tracks the base instead: Swahili, the language the base knows least, gains $10\times$ more than German.³ Our capacity math says a rank-1 adapter would be hopeless here, but it is not.

Why rank 1 might suffice. Our hypothesis treated language as storage with one independent fact per direction. However, our result suggests a language update is high-information yet low-rank. Linguistically, a language has properties a transformer can learn well (eg. word order, morphology), so one small correction could cover more sentences, which carries over to sentences the adapter never saw.

Aya (8 languages). We can stress test the hypothesis further by training a single adapter on eight typologically diverse languages across five scripts. If each language were its own direction, the combined update should need eight. In reality, our effective rank stays near $1.5$. The languages appear to share a cross-lingual subspace, and the combined update remains one correction. The spectrum graph will elaborate on this result.

Super-NaturalInstructions: Massive multitask

In the same spirit, Super-NI can help us test our capacity laws in the massive multi-task setting. Languages might share a subspace, but genuinely different tasks might not. We combine eight task types from Super-NI (arithmetic, NER, summarization, sentiment, POS tagging, QA) into one adapter. Here, the effective rank was $\approx 2$ and did not climb with the number of tasks. In fact, the effective rank fell, potentially as a shared instruction following direction absorbed much of the update. Our NI unrelated tasks are not especially orthogonal in weight space.

PopQA: Long tail facts

The real-world task closest to our synthetic set is PopQA, real Wikidata facts phrased as single-answer questions (“What is the capital of ___?”). Some of its entities are famous and some are obscure, measured by the Wikipedia pageviews of the subject entity. We split the dataset into three popularity bins — head (popular entities), torso (middle), and tail (obscure) — and finetune on each. Our bet is on the tail. An obscure entity barely appears in pretraining, so facts about it should be the closest thing to our random triples, and the adapter should have to store them.

Once again, our hypothesis is negative. Gains are rank-flat and the update stays near one direction in every bin as measured by bits/token (in contrast, random fact memorization sits at effective rank $\approx 14$):

Long-tail factual QA still generalizes. The answers are real entities the base already represents, so the adapter build on existing pathways rather than storing new bits, and one or two directions are enough.

Rank tracks structure, not data

Across single languages, multilingual injection, massive multi-task, and long-tail facts: every real task we measured used an effective rank of 1 to 3, while random facts spread across ~14. Even when we stress test the adapter, such as growing the fact set from 500 to 549k, every real task stays at 1 to 3 regardless of how much data it sees.

Visually, we can observe the spectrum of the updates themselves. Each adapted layer’s $\Delta W$ decomposes by SVD into independent directions, $\Delta W = \sum_i \sigma_i\, u_i v_i^\top$, with $\sigma_i$ the strength of each. We normalize the 16 singular values of each layer to sum to one (only the shape matters), and average over the 65 layers. A spectrum piled onto $\sigma_1$ is a single-direction update; a flat spectrum fills all sixteen:

The spectrum makes the two regimes visible at a glance. Random facts spread their weight across all sixteen directions, while every real task piles onto the first one or two, including the combined 8-language and 8-task adapters.

The practical reading. Once again, we can see that binding constraint is data and base quality, rather than adapter rank, where capability can be bought with more examples or a stronger base.

Previous works. Morris et al. emphasize the difficulty of separating memorization from the rest of a model’s behavior: “a language model prompted to add two numbers can output the answer without having seen the equation before.” PopQA is a case in point here. Obscure facts show a large loss drop that looks like memorization, yet the update is low-rank and generalizing. Their remedy separates the two in the loss, decomposing against a reference model pretrained from scratch; our adapter’s rank separates them in the weights, since memorization fills up the ranks and generalization does not. The rank-flatness itself is an older thread. Aghajanyan et al. found that fine-tuning has a tiny intrinsic dimension, and Schulman et al.’s LoRA Without Regret argues from capacity that rank 1 already exceeds what most post-training needs. Their argument is about the RL signal itself. A policy-gradient update carries only O(1) bits per episode, so a rank-1 adapter already outstrips what there is to store. Aya sits on the other side of that logic and still lands near rank 1.5. Its training tokens carry roughly 170× more information than a rank-1 adapter can even hold, the same 170× as the hypothesis above, so a sparse signal cannot be the reason here. The two results land on the same practical answer for different reasons. One comes from how little information arrives, the other from the shape of what is being learned. The one place the literature sees rank clearly bite, Biderman et al.’s LoRA Learns Less, finds LoRA trailing full fine-tuning on code and math, with the gap closing only as rank grows. This is the information-dense, memorization regime our spectrum picks out.

How to know the smallest rank?

In production, we are always interested in shipping the smallest rank. It is the cheapest to store, to serve, to merge back into the base, and leaves no capacity idle. The usual way to find out is to sweep several ranks for a comparison, which multiplies the number of training runs. The results above point to a cheaper route, given that we have observed adapter rank is fixed by task structure rather than by data volume.

Unfortunately, task structure is hard to gauge. Two things set the rank a task needs: enough directions to express the transformation itself, which is 1 to 3 for every real task we measured, and enough capacity to cover whatever must be stored outright. Real tasks are bound by the first; pure fact memorization by the second. Neither can be read off the frozen base. In our experiments, the rank of the loss gradient at initialization does not predict the rank of the converged update — the update’s rank is a property of the optimization trajectory, not of the initial geometry. Instead of predicting the rank, we can measure it via a quick adapter training run.

A simple method

Training cost is nearly independent of rank: for the 1B base a rank-32 adapter is only about 0.2% of the trainable parameters, so rank 32 and rank 4 are both reasonably cheap. The usual reason to search over rank therefore disappears. We train once at a generous fixed rank $R$ (32 or 64), and recover the rank the task needs from that single run.

Make the target precise. Fix a tolerance $\epsilon$, write $\Delta W^\star$ for a converged rank-$R$ update and $\Delta W^\star_k$ for its rank-$k$ SVD truncation, and let $\mathcal{L}$ be the held-out loss. The rank worth deploying is the smallest that surrenders no more than $\epsilon$ of the achievable loss reduction,

The point is that $r^\star$ is a property of the already-trained $\Delta W^\star$, so no retraining is needed to find it. Per the Eckart–Young–Mirsky theorem, the top-$k$ SVD truncation is the best rank-$k$ approximation of a matrix you can make, and its error is exactly the singular values you threw away, $\lVert \Delta W^\star - \Delta W^\star_k\rVert_F^2 = \sum_{i>k}\sigma_i^2$. As long as the loss varies smoothly with the weights, a small tail means a small change in loss, and the spectrum of one generous run pins down $r^\star$.

Algorithm. Given a base, task data, a “generous” rank $R$, and tolerance $\epsilon$:

Results. To confirm the measured rank is the rank the task needs, we truncate each trained rank-32 adapter to rank $k$ by its rank-$k$ SVD and remeasure the test loss, with no retraining. The regimes separate cleanly. For the three real tasks the loss is already at its floor by rank 1 and stays flat out to rank 32. Interestingly, the truncated adapter for PopQA is even slightly better at rank 1, where discarding directions acts as helpful regularization. A skill’s $\Delta W$ already lives in one or two directions, so its rank-1 approximation reconstructs almost the entire update. Random-fact recall is the opposite: the loss keeps falling all the way to rank 32, since each direction stores more facts. The measured effective rank therefore coincides with the rank the task uses, and whether truncation is free or costly identifies the regime after the fact. The rank trajectory shows the same signal even earlier: a skill’s update collapses toward one direction over training while memorization climbs, so the direction of the trend separates the two well before convergence.

The truncated adapter also holds for decoding, not just test loss. From greedy generation, the rank-1 version reproduces the full-rank outputs almost token for token, and PopQA exact-match is flat from rank 1 (about 0.15, against 0 for the frozen base) while random fact recall again climbs with rank. So the truncation is capable of preserving generation quality, not only its likelihood.

Limitations

• Our experiments are limited to a dense transformer architecture and the layers where LoRA is applied.
• Every adapter keeps the token embeddings and LM head frozen, with LoRA on the attention and MLP linear layers only. For tasks that shift the token distribution, such as a new language, training the embeddings may add capability that our adapter architecture cannot reach. We saw hints of this on the multilingual task, but did not pursue it further.
• We lightly tune learning rates across all experiments, but cannot guarantee that they are optimal.
• Task gains are measured mainly as test loss. We spotchecked greedy generation under truncation and found the rank-1 adapter reproduces the full adapter’s outputs (PopQA exact-match flat from rank 1; fact recall climbing with rank), buthave not run a broad downstream evaluation across all tasks.

Notation

References

Thanks to Claude Opus 4.8 for assistance in making these figures and writing code for the experiments.

• Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models.
• Aghajanyan et al. (2020). Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning.
• Biderman et al. (2024). LoRA Learns Less and Forgets Less.
• Morris et al. (2025). How Much Do Language Models Memorize?
• Allen-Zhu & Li (2024). Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws.
• Busbridge et al. (2025). Distillation Scaling Laws.
• Schulman et al. (2025). LoRA Without Regret. Thinking Machines Lab. thinkingmachines.ai/blog/lora
• Magnusson et al. (2025). DataDecide: How to Predict Best Pretraining Data with Small Experiments. Ai2.
• Singh et al. (2024). Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning.
• Mallen et al. (2023). When Not to Trust Language Models (PopQA).

Citation

If you find this useful, please cite it as:

@misc{nguyen2026loracapacity,
  author       = {Nguyen, Tai},
  title        = {Capacity of a LoRA Adapter (Part 1)},
  year         = {2026},
  howpublished = {\url{https://taidnguyen.github.io/blog/lora-capacity/}},
  note         = {Blog post}
}

Tai Nguyen. “Capacity of a LoRA Adapter (Part 1).” 2026. taidnguyen.github.io/blog/lora-capacity