TX 04.1 · TRANSMISSION · 2026-08-20 · SIGNAL CLEAR

What actually makes a 27B model faster on an 8 GB GPU

RTX 3070 Ti (8 GB) · i7-12700K · 64 GB DDR4-3600 · WSL2 · ~9 min read

A 27B parameter model at Q4_K_M is about 16 GB of weights. My GPU has 8 GB. Every token therefore requires dragging most of the model across the PCIe bus from system RAM, which is why the honest starting number was roughly 1 to 2 tokens per second.

I spent a night measuring what actually helps. Some of it was surprising, one thing was a genuine bug in my setup, and the technique everyone recommends made things worse. All numbers below are from the same fixed prompt set on the same machine.

The setup

Everything runs in WSL2 on Windows. I built llama.cpp from source with CUDA support rather than using a prebuilt binary, because the flags that matter for this problem are not exposed by higher-level wrappers:

cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 \
      -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server llama-cli llama-bench -j 10

CMAKE_CUDA_ARCHITECTURES=86 is the compute capability for Ampere consumer cards. Setting it explicitly avoids building kernels for architectures you do not have.

What I measured

The per-prompt spread mattered more than the averages. On the dense 27B with MTP, a long analytical answer ran at 2.35 tok/s while a short formatted answer ran at 1.17 — the fixed cost of prompt processing dominates when the output is short.

Multi-token prediction is free throughput, if your model has the head

Some recent models ship a multi-token prediction head: an extra block trained to guess the next few tokens. It is normally used during training, but at inference it can act as its own draft model — the target model proposes several tokens and verifies them in a single forward pass.

This matters enormously when you are memory-bandwidth bound. If a token costs one full sweep of 16 GB of weights, and you can verify four tokens in that same sweep, you get close to four tokens for the price of one. The arithmetic is completely different from a GPU-resident model, where you are compute bound and this trick buys much less.

I checked whether my quantized file still had the head by listing its tensors:

python3 -c "
from gguf import GGUFReader
r = GGUFReader('model.gguf')
names = [t.name for t in r.tensors]
print([n for n in names if n.startswith('blk.64')])
"

It did — fifteen tensors in a final block past the 64 transformer layers. Turning it on is one flag:

llama-server -m model.gguf --spec-type draft-mtp

Result: 1.14 → 1.70 tok/s. Roughly 50% more throughput, zero extra memory, one flag. If your model has an MTP head and you are running it partially on CPU, this is the highest-value thing you can do.

Draft-model speculative decoding did not work, twice

The usual advice is to pair a large target model with a small draft model from the same family. I had a 4B model from the same lineage, so it should have been a clean pairing. It failed for two separate reasons, and both are worth knowing.

First, the models were not actually compatible. Loading it as a draft produced:

error loading model hyperparameters:
key qwen35.rope.dimension_sections has wrong array length;
expected 4, got 3

Same architecture family, same tokenizer, different positional-encoding geometry. "Same family" is not sufficient — the draft and target must agree on more than vocabulary.

Second, and more interesting: for mixture-of-experts models the technique is inverted. Published benchmarks on a comparable card show every speculative configuration coming out net-negative against a 35B MoE baseline: 135.7 tok/s baseline, 129–131 with n-gram drafting, 120 with a small draft model. The reason is structural. An MoE routes each token to a different subset of experts, so drafting several tokens ahead pulls several different expert slices through memory. The batched verification that makes speculation profitable on a dense model buys you nothing.

There is also a plain memory cost: on a 7.5 GB budget, a 0.8B draft plus its KV cache takes 0.7 to 1.0 GB straight out of your context window.

The bug: Windows was quietly moving my VRAM into system RAM

This one cost me the most time and is the most likely to affect other people.

Partway through, my numbers collapsed — a configuration that had measured 21 tok/s dropped to 4.45 with more layers assigned to the GPU. Nothing in the model or flags had changed. The GPU reported 7.5 GB free.

The tell came from comparing what the loader claimed against what the driver reported. llama.cpp said it had offloaded roughly 15 GB of layers. nvidia-smi showed 2.2 GB actually resident on the card.

The cause: NVIDIA's CUDA sysmem fallback policy. When an allocation would not fit, the driver silently places it in system RAM and lets it be accessed over PCIe rather than failing. Your program believes it has a GPU allocation. In practice every access crosses the bus. It is not an error, so nothing logs it — you just get a mysterious five-to-ten-times slowdown.

Two settings govern this:

• NVIDIA Control Panel → Manage 3D Settings → CUDA - Sysmem Fallback Policy → Prefer No Sysmem Fallback. This makes CUDA fail loudly instead of degrading quietly. On WSL2 it must be set globally; per-application entries are ignored.
• Hardware-accelerated GPU scheduling (Settings → System → Display → Graphics). With it off, WDDM budgets allocations conservatively and demotes them more readily.

If you take one thing from this post: when a local model is inexplicably slow, compare the loader's claimed offload against nvidia-smi. If the numbers disagree, you are not running on the GPU you think you are.

The other quiet thief: everything else that wants the GPU

Identical configurations measured 21, 8, and 2.6 tok/s across three runs. The variable was not my code — it was a model server that had loaded another model in the background, a screen recorder holding an encoder, and a desktop compositor.

On an 8 GB card there is no slack. Anything else touching the GPU pushes layers off it, and the fall-off is not gradual: below roughly 50% of layers resident, you are effectively running on CPU. Before any benchmark, I now check what else is holding memory:

nvidia-smi --query-compute-apps=pid,process_name,used_memory \
           --format=csv,noheader

Moving my displays to the motherboard's integrated GPU freed roughly 3.5 GB. On a card this size that is around fourteen additional layers, and the discrete GPU still renders games normally — it just hands finished frames to the iGPU for display.

What I would tell someone starting from here

1. Verify you are actually on the GPU. Compare claimed offload against nvidia-smi. Fix sysmem fallback before tuning anything else.
2. Free the VRAM you already own. Move displays to integrated graphics, close other model servers, check for background GPU consumers.
3. Turn on MTP if your model has the head. One flag, no memory cost, meaningful gain when you are bandwidth bound.
4. Prefer a mixture-of-experts model over a dense one at this size. A 30B MoE with 3B active parameters gave me several times the throughput of a 27B dense model, because only a fraction of the weights move per token. Use --n-cpu-moe to keep expert weights on CPU while attention stays on the GPU, and step the number down until VRAM is nearly full.
5. Do not assume speculative decoding helps. On MoE it is usually negative. Measure it.
6. Be honest about the ceiling. A dense 27B on 8 GB is not going to reach interactive speed; the weights simply have to move. If you need 25+ tokens per second, the answer is a smaller model, an MoE, or more VRAM — not another flag.

None of this is exotic. It is mostly finding out where the time actually goes, which on constrained hardware is rarely where the documentation implies.

■ END OF TRANSMISSION · all transmissions · return to descent