Soup is an open-source, Apache-2.0 licensed CLI (soup-cli) for fine-tuning and post-training LLMs from a single YAML config. It was created by Alpamys Makazhan. It handles batch size, GPU detection, and quantization automatically, and can train locally with QLoRA — no SSH, no cloud required.

How does Soup train an 8B model on a 4 GB GPU?

Through 'layer streaming.' The frozen, quantized base model is kept out of VRAM and streamed to the GPU one decoder layer at a time; only the LoRA adapter actually trains. Combined with 4-bit NF4 quantization, this fits Llama-3.1-8B into ~3.32 GB of peak VRAM on an RTX 3050 Laptop 4 GB card.

Is the layer-streaming result trustworthy?

It's unusually well-documented. The streamed run is reported as bit-exact against a normal resident run, independently reproduced on an H100 at the same 3.32 GB, and shipped with a free Colab T4 notebook that caps the process to 4 GB and asserts the streamed model is bit-identical. There's also a Zenodo paper. It is still marked BETA and opt-in via stream_layers: true.

How do I install and run Soup?

pip install "soup-cli[train]" then 'soup init --template chat' and 'soup train'. The bare 'soup-cli' package is a lighter CLI; the [train] extra adds fine-tuning. Requires Python 3.10–3.12.

What are the limitations?

Layer streaming is BETA and opt-in. The headline 119.6 tok/s figure was measured on v0.72.2 before a v0.73.0 correctness fix that cost ~4.8% at 32B and hasn't been re-run on a 4 GB card since. Streaming trades VRAM for repeated weight reads, so it's slower than a fully resident run — the point is feasibility, not peak speed.

Soup: Fine-Tune an 8B LLM on a 4 GB Laptop GPU From One YAML File

By Prahlad Menon Published 2026-09-12 4 min read

Fine-tuning an 8B model has always meant renting a big GPU or owning one. The bottleneck is VRAM: the full base model has to sit in GPU memory during training, and an 8B model in 16-bit doesn’t come close to fitting on a laptop card. Soup — an open-source CLI from Alpamys Makazhan — attacks that assumption directly, and does it from a single YAML file.

The claim that’s making the rounds: fine-tune Llama-3.1-8B on a 4 GB laptop GPU. What makes it worth a post is that the claim comes with receipts.

The trick: layer streaming

A transformer is a stack of decoder layers. During a forward/backward pass you only need one layer’s weights on the GPU at any given moment. Soup’s layer streaming exploits exactly this: the frozen, quantized base model lives in RAM (or on NVMe), and each decoder layer is streamed into VRAM just in time, used, and evicted. Only the small LoRA adapter is actually resident and trainable.

Add 4-bit NF4 quantization on top and the numbers get striking. Measured on an RTX 3050 Laptop 4 GB:

Llama-3.1-8B-Instruct + NF4, LoRA, batch 1, seq 512
3.32 GB peak VRAM, 119.6 tok/s
Bit-exact against a normal resident run
Independently reproduced on an H100 at the same 3.32 GB

The config to turn it on is one line:

# soup.yaml
training:
  stream_layers: true      # base streams out of VRAM; only the adapter trains
  quantization: 4bit       # NF4 — ~4x smaller, so 8B fits a 4 GB card
  batch_size: 4            # bigger batches amortise the weight read
  stream_source: auto      # RAM when it fits, NVMe disk when it doesn't

Why the evidence is better than usual

“Train an 8B on 4 GB” is the kind of headline that’s usually one benchmark away from falling apart. Soup ships the falsification tools alongside the claim:

A free Colab T4 notebook that caps the process to 4 GB, then asserts the streamed model is bit-identical to a normally-loaded one.
A Zenodo paper (Exact Layer Streaming: LoRA Fine-Tuning of an 8B Model on a 4 GB Laptop GPU) with a DOI.
Full benchmark logs in the repo.

“Bit-exact” is the important word. Many memory-saving tricks change the math slightly; Soup’s pitch is that streaming is a pure scheduling change — same gradients, same result, just a different place to keep the weights between uses.

The actual product is the YAML workflow

Layer streaming is the eye-catching feature, but the reason to reach for Soup day to day is that it collapses fine-tuning into three commands:

pip install "soup-cli[train]"
soup init --template chat
soup train

The rest of the pitch is aimed squarely at the “I spent all day fighting infrastructure” problem:

Zero SSH — no logging into a broken GPU box.
One config — a single YAML file.
Auto everything — batch size, GPU detection, quantization are handled.
Works locally — QLoRA on your own GPU, no cloud required.
Multiple backends — transformers and MLX (Apple Silicon), with a live dashboard and soup doctor --config to tell you which settings a backend actually reads.

That last point reflects a maturing project. Recent releases got strict about silent failures: an unknown config key now refuses to load (exit 1) instead of being quietly dropped, and settings that one backend validates but ignores are now named rather than swallowed. Those are the kinds of papercuts that quietly ruin fine-tuning runs, so it’s good to see them treated as bugs.

Caveats worth stating plainly

Layer streaming is BETA and opt-in. It’s not the default path.
Streaming trades VRAM for repeated weight reads — it’s slower than a fully resident run. The win is feasibility on hardware you already own, not peak throughput.
The 119.6 tok/s number was measured on v0.72.2, before a v0.73.0 correctness repair (−4.8% at 32B) and hasn’t been re-run on a 4 GB card since. Treat the exact tok/s as indicative.
Python 3.10–3.12, Apache-2.0.

Why this matters

The interesting shift here isn’t a new model — it’s that the memory wall for training an 8B model is turning out to be a scheduling problem, not a hard physical limit. If a frozen base can be streamed bit-exactly, then “what can I fine-tune?” stops being set by your GPU’s VRAM and starts being set by your RAM and disk plus patience. For anyone teaching, prototyping, or fine-tuning on domain data without a cloud budget, that’s a real democratization step.

Links: