Capacity of a LoRA Adapter (Part 1)

Introduction

Finetuning is a commonly applied paradigm for adapting language models (LLMs) to downstream tasks via specifically designed data. In 2021, Hu et al. observes that this update is approximately low rank, making a full update on model parameters unnecessary.

Specifically, for a weight matrix $W_0 \in \mathbb{R}^{d\times k}$, LoRA replaces the dense update $\Delta W$ with a low-rank factorization,

where $B \in \mathbb{R}^{d\times r}$, $A \in \mathbb{R}^{r\times k}$, and $r \ll \min(d,k)$. The scalar $\alpha$ is a fixed scale set once and not trained ($\alpha = 1$), and dividing it by rank $r$ keeps the size of the update roughly constant as the rank changes. This is a nice proposal that allows one learning rate to work well across $r$ choices.

Since only $A$ and $B$ get trained while $W_0$ stays frozen, LoRA drops the trainable count from $dk$ down to $r(d+k)$ per weight matrix. The result is a small, detachable patch (also called an “adapter”) $\{A, B\}$ pair that conserves memory in training.

A playable animation for LoRA decomposition

Even at the largest rank, the adapter $\{A, B\}$ stays a small fraction of $W_0$. This saving has popularized LoRA to be the canonical post-training method. In many deployments, it is effectively the only method. For example, a serving stack that hot-swaps thousands of task-specific adapters over one base. Whenever you are committed to an adapter, the question that decides whether finetuning will work is a capacity question: since the patch is a sliver of the model, how much can it actually hold?

In this set of experiments, I ask: how much data can a LoRA adapter store, and to what degree can its capacity be linked to the base model?

Morris et al. (2025) - one of my favorite papers this year! - provide a way to quantify such a capacity question. They separate what a model memorizes from what it generalizes by training on random facts, where by memorization can directly be inferred by the number of data bits an adapter can reproduce. Notably, here the author concludes that a full model holds about 3.6 bits of information per parameter.

DataDecide, a suite of 10,000 checkpoints we released with Ai2 last year, can be a nice bed of resources to answer these questions. Across a few model sizes, their step checkpoints, and tasks, we found:

Measuring capacity via Memorization

One way to make capacity concrete is to look at it through memorization: count the bits a model is asked to absorb during finetuning, then ask how many of them actually get observed in the weights. Memorization is less glamorous than generalization, but in a good model the two are often tightly coupled. To get even more complex, model might also learn a way to compose the facts that it holds, making memorization and skills inseparable (Morris et al). A frontier model cannot answer Terminal Bench questions without knowing the syntax of grep/find/cd, and some logical ways to combine them.

The only way to separate Memorization and Generalization is to make the bits maximally ungrokkable.

Setup: To this end, we build a pool of random (entity, relation, value) by sampling from the OLMo tokenizer. 4,000 entities x 800 attributes x 4,000 values give us a total of 3.2 million facts. Every fact holds approximately 12 bit, equating to ~4.8 MB of data. For ease of training, we write them as short sentences with a single-token answer, like so:

Each entity and value is a real word that the OLMo tokenizer encodes as a single token in its vocab, keeping the answer logit in a single position. Given this setup, the only way for the model to complete Attribute 61 of Perry is __ is to have memorized Yugoslav.

Since the value is drawn uniformly from $V = 4{,}000$ candidates, a model that has never seen it can do no better than guess, and naming the right one from scratch costs $\log_2 V \approx 12$ bits. Learning one fact is worth about 12 bits. A model that has learned the fact spends fewer bits to store it, with the savings made possible by the trained weights.

Formally, we can test exactly the unintended-memorization measure suggested in Morris et al. as: $\text{bits} = \log_2 V - \text{NLL}_2(\text{answer} \mid \text{query})$, the uniform prior over the $V$ possible values minus whatever it still gets wrong.¹

The capacity ceiling

To measure how many $bits$ a LoRA adapter can hold, we train each model and adapter rank on a growing set of $D$ facts and read off the bits it memorizes. All experiments stay in bfloat16 and seq=256.

Capacity $C$ is the maximum, over dataset size $D$, of the total bits the trained adapter memorizes:

where the sum runs over the $D$ facts in the training set. Below a critical $D$ the post-trained model rides the diagonal and stores every fact, so bits just track $D$. Past it, storage saturates and then declines as new facts overwrite old ones. That plateau is the capacity, and the two regimes meet at $D_c = C/\log_2 V$.

We visualize the capacity of each base and rank setups via their memorization curves below.

Bits memorized against training-set size for the 530M base, one line per LoRA rank, log-log.

Each curve rides the diagonal while it has room, then peels off at its own ceiling. Intuitively, smaller bases peel sooner, the 20M near ten thousand facts and the 530M past a hundred thousand. At rank 4, the 20M adapter crosses from about 104 kbit at 18k facts to roughly -39 kbit at 400k, having overwritten more than it kept. Pushed far past capacity, the smallest model even does worse than random guessing, where its stored bits turn negative. Schulman et al.’s LoRA Without Regret sees the first half of this shape on real tasks: loss curves track full fine-tuning until a rank-dependent threshold, then fall off as the adapter runs out of capacity. The overshoot into negative territory, though, is ours; in their setting a past-capacity LoRA learns less efficiently rather than regressing.

The capacity law

How can we predict the capacity of a given base and adapter pair before training? Naturally, we can turn to scaling laws. Some dimensions we discuss thus far provide valuable signals: the base parameter count $N$, number of tokens $T$ saw in pretraining, and the adapter’s trainable parameter count $p \approx 2\,r\,d_\text{model} \times (\text{layers} \times \text{matrices})$, with the adapter on every attention and MLP projection. Since DataDecide supplies matched checkpoints across abundant model sizes and pretraining steps, we can fit a scaling law over some (base, rank) cell.

We posit a joint power law, $C \propto N^{a}\, p^{b}\, T^{c}$, and estimate the exponents by least squares in log space,

We can visualize the performance of fitting on different variable combinations:

Our fit is done on models up to 530M, with 1B held out. The $N \cdot p$ fit uses the 25 finals (≤530M, ranks 1–16); the $p \cdot T$ and $N \cdot p \cdot T$ fits add the pretrain step checkpoints (all six sizes, ranks 1/4/8/16). With any fit involving T, we use 101 points.

Two findings stand out. First, the exponent on $p$ lands near one in every view ($0.97$ to $1.08$): capacity is essentially linear in the adapter’s trainable parameters. Second, one variable on model quality is adequate. With pretraining tokens in the fit, the exponent on $N$ turns slightly negative, so the base’s raw size adds no predictive power on top:

Dropping $T$ lets size stand in as its proxy at nearly the same quality,

In “Distillation Scaling Laws”, Busbridge et al. also treat base size and pretraining tokens as two handles on the base’s quality, where teacher size and teacher tokens reach the student only through the teacher’s loss. It seems we can use $T$ when the pretraining budget is known, but otherwise $N$ can stand in: in DataDecide, models are trained to roughly 100 tokens per model parameter, about 5x Chinchilla-optimal.

Is the base-quality term causal, or is $N$ merely correlated with it? A bigger base is also a more heavily pretrained one, so the clean test holds each base and adapter fixed and varies only how long the base was pretrained, walking DataDecide’s checkpoints from random initialization to the final model (plotted as capacity per adapter parameter, $C/p$, to put all sizes on one scale).

Capacity rises monotonically with pretraining, at every size and rank. A random base sits at the injection floor, near 0.01 bits per parameter, because it has no representations to remap; pretraining lifts it severalfold, from about 0.05 at 20M to roughly 0.2 at the mid sizes and 0.4 at 1B by the end of the sweep. Holding size and parameter count fixed while adding tokens still raises capacity, so the base-quality term in the law is causal, not just a correlation with size. The lift shows up across every rank measured (1, 4, 8, 16), not just one.

When fit on models up to 530M, the law predicts 1B’s capacity within 0.8 to 1.4 times measured (the held-out $R^2 = 0.96$). Notably, a fit on $p$ alone collapses on the held-out 1B ($R^2 = 0.45$). Adding either $N$ or $T$ as an axis restores our predictability. I have found that a fit on <=150M can work almost equally well ($R^2 = 0.92$ for predicting 1B), so a cheap budget might suffice in practice.

With the law, we can quantify precisely the capacity of an adapter rank. Because the exponent on $p$ is essentially one ($p^{1.08}$), dividing the fit through by $p$ gives the bits per LoRA parameter,

Since the exponent on $p$ is small, density is mostly a property of the base model instead of the adapter size: at 530M it sits between 0.13 and 0.20 across ranks 1 to 16, while the base moves it from about 0.05 at 20M to 0.4 at 1B. Specifically:

Thus, in our suite, a LoRA parameter holds about 0.05 to 0.4 bits per trainable parameter.

How do these densities compare to full training? Morris et al. train GPT-family models from scratch on random data, the same protocol we borrow here, and read a plateau of about 3.6 bits per parameter (also in bfloat16). Allen-Zhu and Li get at the number differently: they train on synthetic biographies and count how many facts can be probed back out, landing near 2 bits per parameter. Against either reading, a LoRA is roughly an order of magnitude less dense than the weights it patches.

Even at 1B the density sits well below the full training ceiling, but a low density is not a weakness in itself. Thinky’s blog post corroborates with the conclusion that a LoRA keeps pace with full finetuning until the finetuning set asks for more bits than the adapter can hold. In the next section, we examine more closely where the capacity boundary falls from the perspective of recall.

From capacity to recall

Our $bits$ and capacity ceiling for LoRA have closely followed Morris’s framing. In practice, we might want to also gage model’s recall ability given that it has been train on a set of $D$ facts. Here, capacity governs both storage (our “bits” defined in previous session) and recall – simply asking how many facts model can guess correct.

For a single base and rank, both stored bits and recall move with dataset size in units of capacity $D_c = C/\log_2 V$. Below $D_c$ the adapter recalls almost every fact and stored bits rise; around $D_c$ storage saturates at $C$ and recall bends over together. Storage stops rising at the same dataset size where recall starts falling, sharing capacity as the yardstick for the size of the training set.²

In the following plot, we collapse our family onto one curve by dividing each curve’s dataset by its capacity in facts, $D_c = C/\log_2 V$.

Every memorization curve, rescaled by its own capacity in facts $D_c = C/\log_2 V$. The sigmoid is fit on the ≤530M cells alone. Shading marks the regimes: store ($D < D_c$), lose ($D_c$ to $5D_c$), collapse ($> 5D_c$).

Recall stays near one before crossing $D_c$, halves when passing it, and is gone by about $5\,D_c$. The same behavior applies at almost every size and rank, with a single shared steepness $k \approx 2.3$.

By substituting in capacity, we can fit Recall as function of the base, the adapter, and the dataset directly,

Before training, we can – similar to scaling law – apply the fact set size to this equation. Below that $C(N,p)/\log_2 V$, the adapter keeps everything. Above it, it will start trading old facts for new.

Real-world post-training

The synthetic law makes rank buy capacity: $D$ structureless facts force an update spanning on the order of $D$ directions, so a bigger adapter stores more. Does that carry over to real fine-tuning? Does a task with more data, or more information-dense data, need more rank? To find out we ran a spread of real tasks and, for each, measured two things: the reduction in test loss each LoRA rank buys, and the effective rank of the update $\Delta W = BA$ the adapter actually learns. The synthetic random facts are the memorization control at one extreme.

We evaluate on the following tasks:

• random facts: the synthetic $(\text{entity}, \text{relation}, \text{value})$ triplets defined earlier, with each fact worth 12 bits and no shared structure to exploit.
• Aya: multilingual instruction following, teaching an English base model a language it has barely seen. Knowledge injection, the high-information extreme.
• PopQA: real Wikidata facts phrased as questions, split by entity popularity. Very close to our synthetic set and is fully grounded.
• Super-NaturalInstructions: a pool of genuinely different instruction task types (translation, arithmetic, NER, summarization, sentiment, and more).

Setup. Every adapter uses LoRA on all linear layers (attention and MLP), trained with each task’s own objective. We read out two things: the reduction in test loss (held-out negative loglikelihood) over the frozen base at each rank, and the effective rank of $\Delta W = BA$: $\exp\big(-\sum_i p_i \log p_i\big)$ over its normalized singular values $p_i = s_i/\sum_j s_j$, the number of directions carrying real weight (median over the 65 adapted matrices). A low effective rank means one shared transformation; a high one means many independent directions.

Aya: injecting a new language

The first task is multilingual instruction tuning on the Aya Collection. This is an information-heavy task, since DataDecide is primarily English.

Hypothesis (from the capacity law, before running). Treat the task as memorization. A new language costs the base roughly 4 bits per token, so each rank’s capacity converts to a token budget — and we train far past all of them:

If the adapter stored language the way it stores random facts, rank 1 would collapse and rank 16 would pull far ahead.

Reality (measured). Neither happens. Rank does nothing; what moves the gain is how unfamiliar the language is to the base (reduction in test loss, nats/token, at 25.6k examples):

Rank 1 matches rank 16 within $0.03$ nats in all eight cells, even at 200× its supposed storage budget. The gain tracks the base instead: Swahili, the language the base knows least, gains $10\times$ more than German. (Hindi only looks familiar because Devanagari fragments into many cheap tokens; its gain is still rank-flat.) The storage meter says a rank-1 adapter should be hopeless here. It is not.

Why rank 1 suffices even here. Our hypothesis treated the task as storage: one independent fact per direction, so $D$ facts need a rank-$D$ update and capacity is the wall. That is true for random facts, and only for random facts.

A real task asks for one transformation, shared by every example. Learning a language is closer to learning to speak than to memorizing a phone book. The adapter that produces Swahili is a single low-rank correction to a base that already has the general machinery, and it generalizes to sentences it never saw. Ten times more data does not grow that update’s rank; it just estimates the same update better. The training tokens are never stored. Information is not rank: a task can carry far more bits than another yet need fewer directions, because what matters is whether the update is one shared transformation or a pile of independent facts.

Pooling eight languages. We then tried to break the picture. If one language is one direction, eight should need eight. We trained a single adapter on eight typologically diverse languages across five scripts. Effective rank stayed near $1.5$, whether one language or eight. The languages share a cross-lingual subspace, so even the eight-way update is one correction.

Super-NaturalInstructions: pooling unrelated tasks

Languages might share a subspace, but genuinely different tasks should not. We pool eight task types from Super-NaturalInstructions (arithmetic, NER, summarization, sentiment, POS tagging, QA) into one adapter. Effective rank was $\approx 2$ and did not climb with the number of tasks — if anything it fell, as a shared instruction-following direction absorbed more of the update. Even unrelated tasks share the follow-the-format structure, so they are not orthogonal in weight space.

PopQA: real facts from the long tail

The last objection is real memorization. We take PopQA, real Wikidata facts phrased as questions, and split it by entity popularity, expecting obscure long-tail facts to behave like our synthetic ones. They did not (gains in nats/token; compare memorization’s effective rank of $\approx 14$):

Long-tail factual QA still generalizes. The answers are real entities the base already represents, so the adapter re-aims existing pathways rather than storing new bits, and one or two directions are enough.

Rank tracks structure, not data

Across single languages, pooled languages, pooled task types, and long-tail facts: every real task we measured used an effective rank of 1 to 3, while random facts spread across ~14. The task sets the rank, not the data budget. Growing the fact set from 500 to 549k pushes the memorization rank toward the ceiling; every real task stays at 1 to 3 no matter how much data it sees.

The cleanest view of this is the spectrum of the update itself. Each adapted layer’s $\Delta W$ decomposes by SVD into independent directions, $\Delta W = \sum_i \sigma_i\, u_i v_i^\top$, with $\sigma_i$ the strength of each. We normalize the 16 singular values of each layer to sum to one, so only the shape matters, and average over the 65 layers. A spectrum piled onto $\sigma_1$ is a one-direction update; a flat spectrum fills all sixteen:

This lines up with Morris et al., who argue you cannot read memorization off a model’s behavior. Producing a fact, or a large likelihood gain, is equally consistent with generalization: “a language model prompted to add two numbers can output the answer without having seen the equation before.” PopQA is a case in point, where obscure facts show a big drop in test loss that looks like memorization but is a low-rank, generalizing update. Their remedy separates the two at the loss level, by decomposing memorization against a reference model. The adapter’s rank separates them in the weights, since memorization fills it and generalization does not. These are two lenses on the same distinction, and it is a real one.

The practical reading. Across these tasks the binding constraint is data and base quality, not adapter rank. Capability is bought with more examples or a stronger base, and among LoRA ranks, rank 1 is already enough, since more rank does not help. The synthetic sweeps set the rank you would need to memorize a dataset outright, and real fine-tuning, being low-rank, sits far below that ceiling. Rank becomes the constraint only when a task looks like memorization. This is not ours alone. It traces back to Aghajanyan et al.’s finding that fine-tuning has a tiny intrinsic dimension, and it matches Schulman et al.’s LoRA Without Regret, which argues from capacity that rank 1 already exceeds what most post-training needs and that rank only bites once the adapter runs out of capacity. The one place the literature sees rank clearly matter, Biderman et al.’s LoRA Learns Less on code and math, is exactly the information-dense, memorization-like regime our spectrum picks out.

How to know the smallest rank?

The smallest rank that does the job is the one worth shipping. It is the cheapest to store, to serve, and to merge back into the base, and it leaves no capacity idle. The usual way to find it is to sweep several ranks and compare, which multiplies the training cost for a number you only use once. The results above point to a cheaper route, because rank is fixed by task structure rather than by data volume. Picking a rank then comes down to two quantities: how many bits a rank-$r$ adapter can store, and how many the task needs it to.

Capacity is what a rank-$r$ adapter can hold. It stores about $b\,p(r)$ bits, where $p(r)$ is its trainable-parameter count and $b\approx 0.05$–$0.4$ is the bits stored per parameter, fixed by the base. This is the capacity law of the previous section.

Requirement is what the task needs to store: its incompressible content given the base, the part that cannot be folded into a transformation the base already computes. A skill such as a language, instruction-following, or factual QA is one such transformation re-aimed, so its requirement is a direction or two and does not grow with the number of training examples. Memorizing $N$ arbitrary facts requires $N\log_2 V$ bits, which does not compress and grows with the dataset. Real tasks fall between these limits.

The appropriate rank is the smallest $r$ at which capacity covers the requirement:

For a skill the right-hand side is small and $r^\ast\approx 1$; for structureless memorization it equals the dataset entropy and $r^\ast$ grows with the data.

The difficulty is estimating the requirement, which is not recoverable from the frozen base. In our experiments the rank of the loss gradient at initialization does not predict the rank of the converged update. The update’s rank is a property of the optimization trajectory, not of the initial geometry, so it is not available before training.

A simple method

Training cost is nearly independent of rank: for the 1B base a rank-32 adapter is about 0.2% of the trainable parameters, so rank 32 and rank 4 cost essentially the same. The usual reason to search over rank therefore disappears. We train once at a generous fixed rank $R$ — 32 to 64 covers every task here, and real tasks need far less — and recover the rank the task needs from that single run.

Make the target precise. Fix a tolerance $\epsilon$, write $\Delta W^\star$ for a converged rank-$R$ update and $\Delta W^\star_k$ for its rank-$k$ SVD truncation, and let $\mathcal{L}$ be the held-out loss. The rank worth deploying is the smallest that surrenders no more than $\epsilon$ of the achievable loss reduction,

The point is that $r^\star$ is a property of the already-trained $\Delta W^\star$, so no retraining is needed to find it. By the Eckart–Young–Mirsky theorem the top-$k$ truncation $\Delta W^\star_k$ is the optimal rank-$k$ approximation of $\Delta W^\star$, and it differs from it by exactly the discarded tail, $\lVert \Delta W^\star - \Delta W^\star_k\rVert_F^2 = \sum_{i>k}\sigma_i^2$. Where the loss varies smoothly with the weights, a small tail means a small change in loss, so the spectrum of a single generous run pins down $r^\star$.

Procedure. Given a base, task data, a generous rank $R$, and tolerance $\epsilon$:

There is no second training run. The shipped adapter is a truncation of the one you already trained, obtained by an SVD in seconds, not a fresh model retrained at $r^\star$. Training happens once, at rank $R$.

Validation. To confirm the measured rank is the rank the task needs, we truncate each trained rank-32 adapter to rank $k$ by its rank-$k$ SVD and re-measure the test loss, with no retraining. The regimes separate cleanly. For the three real tasks the loss is already at its floor by rank 1 and stays flat out to rank 32, so truncating to a single direction costs nothing; on PopQA it is even slightly better at rank 1, where the discarded directions were fitting noise. A skill’s $\Delta W$ already lives in one or two directions, so its rank-1 approximation reconstructs almost the entire update, not a fragment of it. Random-fact recall is the opposite: the loss keeps falling all the way to rank 32, since each direction stores more facts. The measured effective rank therefore coincides with the rank the task uses, and whether truncation is free or costly identifies the regime after the fact. The rank trajectory during training carries the same signal earlier: a skill’s $\Delta W$ collapses toward a single direction while memorization rises in rank, and the sign of that trend separates the two well before convergence.

The procedure needs one training run and no rank search, and the deployed adapter is as small as the task permits. When measurement is not worth the trouble, two shortcuts suffice. Structured, generalizing tasks want rank 1 to 8 (we measure 1 to 3, so 8 is a safety margin). Deliberate memorization should be sized from the budget, $r \gtrsim (N\log_2 V)/(b\cdot\text{params-per-rank})$.

Three things follow from reading rank this way. The effective rank of $\Delta W$ is a memorization detector taken from the weights alone: it is high when a task stores data and low when it generalizes, so a rank that climbs during fine-tuning flags memorization while it happens, with no reference model needed. Because truncation is lossless for a low-rank update, the rank sweep is avoidable, and one generous run plus a truncation gives the adapter to ship. And truncation is occasionally better than the original, as on PopQA, when the directions it removes were fitting noise.

Limitations

• Our experiments are limited to a dense transformer architecture and the layers where LoRA is applied.
• Every adapter keeps the token embeddings and LM head frozen, with LoRA on the attention and MLP linear layers only. For tasks that shift the token distribution, such as a new language or script, also adapting the embeddings may add capability that a linear-only adapter cannot reach. We saw hints of this on the multilingual task and did not pursue it further.
• We lightly tune learning rates across all experiments, but cannot guarantee that they are optimal.
• The rank a task needs is set by its incompressible complexity, which we currently measure (from the converged or trajectory rank) rather than predict a priori. A predictor that reads it off the base and data without training would complete a full rank law $r^\ast = f(\text{complexity}, \text{base})$.
• Task gains are measured as test loss, not task accuracy. The reductions on PopQA and Aya are large and well above noise, but we have not confirmed that truncation preserves exact-match accuracy or generation quality, only likelihood.

Notation

References

Thanks to Claude Opus 4.8 for assistance in making these figures and writing code for the experiments.

• Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models.
• Aghajanyan et al. (2020). Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning.
• Biderman et al. (2024). LoRA Learns Less and Forgets Less.
• Morris et al. (2025). How Much Do Language Models Memorize?
• Allen-Zhu & Li (2024). Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws.
• Busbridge et al. (2025). Distillation Scaling Laws.
• Schulman et al. (2025). LoRA Without Regret. Thinking Machines Lab. thinkingmachines.ai/blog/lora
• Magnusson et al. (2025). DataDecide: How to Predict Best Pretraining Data with Small Experiments. Ai2.
• Singh et al. (2024). Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning.
• Mallen et al. (2023). When Not to Trust Language Models (PopQA).

Citation

If you find this useful, please cite it as:

@misc{nguyen2026loracapacity,
  author       = {Nguyen, Tai},
  title        = {Capacity of a LoRA Adapter (Part 1)},
  year         = {2026},
  howpublished = {\url{https://taidnguyen.github.io/blog/lora-capacity/}},
  note         = {Blog post}
}

Tai Nguyen. “Capacity of a LoRA Adapter (Part 1).” 2026. taidnguyen.github.io/blog/lora-capacity