LoRA vs QLoRA vs full fine-tuning comes down to what gets frozen and what gets trained. Full fine-tuning updates every parameter, so memory has to hold the weights, the gradients, and the optimizer state together. For a 7 billion parameter model that totals about 112 GB, before any activations. LoRA freezes the pretrained weights instead and trains small rank decomposition matrices injected into each layer. The base then costs only 14.0 GB, since it carries no gradients or optimizer state of its own. QLoRA goes further still. It backpropagates through a frozen 4-bit quantized base into those same LoRA adapters, so the base drops to 3.5 GB. Because the low-rank update merges back into the original weights, neither LoRA nor QLoRA adds inference latency. The QLoRA paper also finetuned a 65 billion parameter model on a single 48GB GPU this way, and its Guanaco models reached 99.3 percent of ChatGPT’s performance level on the Vicuna benchmark after 24 hours on one GPU.
Fine-tuning an open-weights large language model, or LLM, used to mean one thing. Every parameter got updated. So memory paid the price. That price scales directly with model size, so a 7 billion parameter model already strains a single consumer GPU.
Low-Rank Adaptation, or LoRA, changed that math. It freezes the base model and trains a small set of extra weights instead. QLoRA pushes the idea further, by quantizing that frozen base to 4-bit precision first.
This guide works through the mechanism behind each method, then the memory it actually needs. A 7B model anchors every number, so you can check the arithmetic yourself. Before committing to any of these, it helps to ask a different question first. Is fine-tuning even the right tool? Our Retrieval-Augmented Generation vs Fine-Tuning guide answers that directly.
Why Full Fine-Tuning Gets Expensive
Training a neural network needs more than the weights themselves. Backpropagation also needs gradients, one per weight, computed every step. Adam, the most common optimizer, then adds two more numbers per weight: a running mean and a running variance.
Mixed-precision training makes this worse before it makes it better. Weights and gradients run at FP16 for speed, but a master copy stays at FP32 for stability. Adam’s two moment buffers also stay at FP32.
Add it up for a 7 billion parameter model, and the totals get large fast. Figures below use decimal gigabytes throughout, where 1 GB equals 1,000,000,000 bytes. They cover weights, gradients, and optimizer state only. Activation memory is excluded, though.
| What is stored | Bytes per parameter | For 7B |
|---|---|---|
| FP16 weights | 2 | 14.0 GB |
| FP16 gradients | 2 | 14.0 GB |
| FP32 master weights | 4 | 28.0 GB |
| Adam first moment | 4 | 28.0 GB |
| Adam second moment | 4 | 28.0 GB |
| Total | 16 | 112.0 GB |
That is 112 GB before a single activation is stored. These techniques apply to neural networks broadly, not only language models. Our Machine Learning vs Deep Learning guide covers that distinction in more depth.
LoRA and QLoRA both attack this total directly. Freeze most of it, so most of the cost disappears with it.
Full Fine-Tuning
Every parameter in the model gets updated here. Nothing stays frozen, so each weight carries its own gradient and its own Adam optimizer state.
That is the full 112 GB total from the section above, for a 7B model. Scale the model up, so that number scales with it, linearly.
Full fine-tuning still earns its place, though. When a task differs sharply from the pretraining data, updating every weight gives the model the most room to adapt. Large-scale continued pretraining, or building a new base model outright, both call for it.
The tradeoff is blunt. You need enough GPU memory to hold the full total, or several GPUs to shard it across.
How LoRA Works
LoRA freezes the pretrained weights and leaves them untouched. Training happens somewhere else entirely: a pair of small matrices injected into each layer of the Transformer.
Those two matrices are called rank decomposition matrices. Multiplied together, they form a low-rank update, added on top of the frozen weight. For a weight matrix sized d by k, the adapter needs only r(d + k) trainable values. In contrast, the base matrix itself holds the full d times k values.
The rank r sets how large that update can be. A smaller r means fewer trainable values. A larger r means more capacity, and more of them.
The original paper, Hu et al., 2021 from Microsoft, tested this against GPT-3 175B fine-tuned with Adam. LoRA cut trainable parameters by 10,000 times, and GPU memory by 3 times, against that full fine-tuning baseline. On RoBERTa, DeBERTa, GPT-2, and GPT-3, it performed on par with or better than full fine-tuning. Still, it did that with far fewer trainable parameters.
Inference is the other half of the story. The low-rank update can merge straight back into the original weight matrix. Because of that, LoRA adds no extra latency at inference time. That merge is what separates LoRA from earlier adapter methods. Those, instead, left a small extra layer in the forward pass permanently.
How QLoRA Adds Quantization
QLoRA, short for quantized LoRA, starts from the same low-rank idea. Then it adds one more constraint: the frozen base itself gets quantized to 4-bit precision.
Gradients still flow through that frozen base during training. They just flow through a base held in 4-bit form, into LoRA adapters kept at higher precision. Only those adapters then get updated.
Three innovations make that work. The first is 4-bit NormalFloat, or NF4, a data type built for weights that follow a normal distribution. The second is Double Quantization, which quantizes the quantization constants themselves, shaving off further memory. The third is Paged Optimizers, which absorb memory spikes during training instead of crashing on them.
Dettmers et al., 2023, presented QLoRA at NeurIPS. Their headline result finetuned a 65 billion parameter model on a single 48GB GPU. It did that while preserving full 16-bit finetuning task performance.
The resulting model family, Guanaco, then reached 99.3 percent of ChatGPT’s performance level on the Vicuna benchmark. That took only 24 hours of finetuning on a single GPU.
Quantization here is doing real work, not just saving disk space. Our Quantization vs Distillation guide covers what lower precision changes inside a model more broadly.
LoRA vs QLoRA vs Full Fine-Tuning: Comparison Table
| Aspect | Full fine-tuning | LoRA | QLoRA |
|---|---|---|---|
| What gets updated | Every weight in the model | Small adapter matrices only | Small adapter matrices only |
| Base weight precision | FP16 (mixed precision) | FP16, unchanged | 4-bit NormalFloat (NF4) |
| Base weights frozen | No | Yes | Yes |
| Trainable parameter share (7B, r=8) | 100% | About 0.06% | About 0.06% |
| Optimizer state size | Full, matches every weight | Adapter-only, tiny | Adapter-only, tiny |
| Memory for a 7B base | 112.0 GB total training memory | 14.0 GB, base only | 3.5 GB, base only |
| Output artefact | A full new model checkpoint | A small adapter file, plus the base | A small adapter file, plus the 4-bit base |
| Task switching | A full model copy per task | Swap adapters, keep one base | Swap adapters, keep one base |
| Inference latency | Baseline | None added; merges into the base | None added; merges into the base |
| Training time | Longest, full backward pass over all weights | Faster, far fewer trainable weights | Slower than LoRA, from quantize and dequantize overhead |
| Hardware reach | Needs the most GPU memory | Fits meaningfully smaller GPUs | Fits a 65B model on one 48GB GPU |
| Quality versus full fine-tuning | Reference point by definition | On par or better on tested models | Preserves full 16-bit finetuning performance |
| When it is the right choice | Ample hardware, or a task unlike pretraining | Constrained memory, need to keep the base | Even tighter memory, largest models |
| Main limitation | Cost of weights, gradients, and optimizer state | Base still sits in memory at FP16 | Compute overhead from 4-bit dequantization |
Worked Example: Fine-Tuning a 7B Model
Take a 7 billion parameter model as the running example. Every figure follows the same decimal-GB convention from earlier, where 1 GB equals 1,000,000,000 bytes.
Full fine-tuning still needs the full 112 GB from the table above. Nothing here is new; it is the same weights, gradients, and optimizer state, just applied to one specific model size.
LoRA changes the picture completely. The base stays frozen at FP16, so it costs 14.0 GB. Instead, it carries no gradients or optimizer state of its own. Only the small adapter matrices need gradients and an optimizer, and those are tiny by comparison.
QLoRA drops the base further still. Stored in 4-bit, that same base costs only 3.5 GB. That is a 4x cut from LoRA’s 14.0 GB base, just from the switch to 4-bit.
Now size the adapter itself. Take a Llama-style 7B model: 32 layers, hidden size 4096. Adapters sit on the query and value projections, as in the original LoRA paper.
At rank r = 8, each matrix needs r(d + k) trainable values. That is 8 times (4096 + 4096), or 65,536 values. Multiply that across 32 layers and 2 matrices per layer, so the total comes to 4,194,304 trainable parameters. Against 7 billion total parameters, that is about 0.06 percent.
Double the rank to r = 16, so the count doubles too. That gives 8,388,608 trainable parameters, about 0.12 percent of the model.
That is the number worth sitting with. You train roughly four million values, not seven billion.
Choosing the Rank
Rank r controls how expressive the LoRA update can be. It sets the dimensionality of the low-rank subspace the adapter is allowed to use.
A low rank keeps the adapter small and fast to train. Fewer trainable values also mean less memory for the adapter’s own gradients and optimizer state.
Push the rank higher, though, and the adapter gains more room to represent complex changes. That extra capacity moves the update closer to what full fine-tuning could express, at the cost of more trainable parameters.
Still, even a large rank stays small next to the base model. Doubling r from 8 to 16 only doubles a number that started at roughly 0.06 percent of 7 billion.
The right rank depends on the task. A narrow, simple adaptation rarely needs much capacity. A task that asks the model to shift its behavior substantially benefits from more room to move.
When to Use Which
Full fine-tuning fits when hardware is not the constraint. A task far outside the pretraining distribution, or a new base model itself, both justify updating every weight.
LoRA fits when the base model already sits comfortably in memory at FP16. It also trains fast, keeps one shared base, and lets you swap adapters between tasks without duplicating that base.
QLoRA fits when even the FP16 base will not fit. A 65 billion parameter model on a single 48GB GPU is only possible because the base drops to 4-bit first.
Hardware is the real constraint behind all three choices. Our GPU vs TPU vs NPU guide covers the accelerators these workloads actually run on.
None of these methods, though, is universally correct. Match the method to the memory you have, and the gap between your task and the model’s original training.
Interview Questions
Frequently Asked Questions
Wrapping Up
LoRA vs QLoRA vs full fine-tuning comes down to what stays frozen. Full fine-tuning updates everything. So it pays for that in gradients and optimizer state. LoRA freezes the base and trains a tiny adapter instead. QLoRA freezes that same base in 4-bit form first, then trains the same kind of adapter on top.
Keep the core numbers in mind. A 7B model costs 112 GB to fully fine-tune. As a frozen LoRA base it costs 14.0 GB, and as a frozen QLoRA base only 3.5 GB. The adapter itself, at rank 8, still adds only about 4.2 million trainable values.
None of these methods is wrong, exactly. Match the method to the memory in front of you. Match it, too, to how far the task sits from the model’s original training.
Related reading on DiffStudy:
- Retrieval-Augmented Generation vs Fine-Tuning
- Quantization vs Distillation
- GPU vs TPU vs NPU for AI Workloads
- Machine Learning vs Deep Learning
- CS Fundamentals hub
