Kimi K3 in C: A 2.78 Trillion Parameter Model on 8GB RAM

This shouldn’t be possible. A 2.78 trillion parameter model. Running on a single CPU. 8GB of RAM. No GPU.

kimi-k3-in-c does exactly that: 176KB of pure C99 code that runs Kimi K3 inference by streaming Mixture-of-Experts from NVMe, keeping only what’s needed in memory.

2.78T parameters
1.56 TB checkpoint on disk
8.24 GB peak RAM (measured)
176 KB entire engine
0 GPUs

GitHub: FareedKhan-dev/kimi-k3-in-c

How Is This Possible?

Kimi K3 is a Mixture-of-Experts model. Only 16 of 896 experts activate per token. The trick: stream experts from disk instead of loading them all into RAM.

The key insight:

• Dense “trunk” layers stay in memory (as much as your RAM allows)
• The 1.45TB of routed experts are never resident—they’re multiplied straight from their packed 4-bit form on disk
• Active experts get loaded on-demand via an LRU cache
• Everything else stays on NVMe

The result? Byte-identical output whether you give it 8GB or 224GB. More RAM = faster (better cache hit rate), but the answer never changes.

The Numbers

From an 8GB laptop:

$ ./bin/k3 ~/k3model --trunk ~/k3trunk --preset laptop \
    --tok ~/k3model --prompt "The capital of France is" --gen 8 --incremental

--- generated text ---
 Paris.",
+ "The Eiffel
----------------------
8 tokens in 261.5 s, 32.69 s/token average
PEAK RSS for the whole run: 8.24 GB

From a 128GB workstation:

$ ./bin/k3 ~/k3model --trunk ~/k3trunk --preset server \
    --tok ~/k3model --prompt "def fibonacci(n):" --gen 28 --incremental

--- generated text ---
 if n <= 1:
    return n
  else:
    return fibonacci(n-1) + fibonacci
----------------------
28 tokens in 299.3 s, 10.69 s/token average
PEAK RSS for the whole run: 127.92 GB

Slow? Yes. But it’s a 2.78 trillion parameter model running on commodity hardware.

The Four Reductions

The paper/docs explain four key decisions that take this from cluster-only to laptop-viable:

1. Experts ship at 4-bit — Already quantized in the checkpoint, multiplied directly from packed form
2. KDA (Key-Difference Attention) — Attention with memory that never grows
3. MLA (Multi-head Latent Attention) — One latent instead of 96 heads
4. Trunk streaming — Memory budget becomes a dial, not a floor

Requirements

The gate is storage: you need 1.7TB free (1.56TB checkpoint + 109GB packed trunk).

Component	Requirement
OS	Linux x86-64
CPU	AVX2 + FMA (no AVX-512 needed)
RAM	8GB minimum, more is faster
Storage	~1.7TB on fast local disk
Toolchain	GCC ≥9 or Clang ≥10

Quick Start

You can verify the engine matches its reference without downloading the 1.56TB checkpoint:

git clone https://github.com/FareedKhan-dev/kimi-k3-in-c.git
cd kimi-k3-in-c

make -j          # Seconds—7 C files
make test        # Under a minute

# Output ends with:
# VERDICT: ENGINE MATCHES THE REFERENCE EXACTLY
# ALL WEIGHTLESS TESTS PASSED

For the full model, you’ll need to download the checkpoint and pack the trunk:

export HF_TOKEN=hf_your_token_here
./scripts/download-model.sh ~/k3model    # 1.56TB, resumable
./scripts/pack-trunk.sh ~/k3model ~/k3trunk  # ~4 minutes

Why This Matters

This is a proof of concept that frontier models can run on consumer hardware through clever engineering:

• No framework dependencies — Pure C99, compiles anywhere
• Deterministic output — Same answer at any memory budget
• Open implementation — Every kernel, the streaming cache, safetensors reader, tokenizer

The 32 seconds/token on a laptop is obviously not practical for real use. But it demonstrates that the hardware barrier to running frontier models is lower than we thought—it’s a storage problem, not a compute problem.

---

GitHub: FareedKhan-dev/kimi-k3-in-c
License: Apache-2.0
Model: Kimi K3 (base model, no chat template)

---

Frequently Asked Questions

Is this actually usable for real work?

Not really. At 32 seconds per token on a laptop (10 seconds on a high-RAM workstation), it’s more a proof of concept than a practical tool. The value is demonstrating that frontier MoE models can run on consumer hardware at all, and providing a clean C implementation for study.

Why is it so slow?

The bottleneck is disk I/O. Every token requires loading experts from NVMe. More RAM means better cache hit rates (experts stay resident longer), which is why a 128GB machine is 3× faster than an 8GB machine. An NVMe SSD is essential—spinning disks would be unusably slow.

Does more RAM change the output?

No. The output is byte-identical regardless of memory budget. More RAM just means fewer disk reads (better cache hits), so generation is faster. The answer never changes.

Why no GPU support?

The goal was proving minimum-viable inference on the most constrained hardware. GPU support would defeat the purpose of demonstrating that a 2.78T model can run on a laptop with 8GB RAM. The engine is also designed for simplicity—176KB of C99 with no dependencies.

Can I use this for the instruct/chat model?

This runs the base model only. There’s no chat template. Output after your prompt is continuation, not reply. The instruct model would require additional implementation.

How does the expert streaming work?

Kimi K3 has 896 experts but only activates 16 per token. The engine maintains an LRU cache of recently-used experts. Cache hits use resident memory; cache misses read from NVMe. The “trunk” (dense layers) stays pinned in RAM to whatever depth your budget allows. Experts are never fully resident—they’re multiplied directly from their 4-bit packed form on disk.

What's the minimum storage requirement?

About 1.7TB total: 1.56TB for the checkpoint (96 safetensor shards) and 109GB for the packed trunk. The trunk should go on your fastest disk since it’s read repeatedly during inference.

Can I run this on macOS or Windows?

Currently Linux x86-64 only. The code uses Linux-specific APIs (O_DIRECT, posix_memalign, getrusage). The tokenizer and config reader are portable C99, but the inference engine needs Linux.