AirLLM: Run 70B LLMs on a Single 4GB GPU (Fast Look)

Ever been told your 4GB GPU can’t run anything serious? Yeah, me too. But every local-LLM guide says “you need at least 16GB VRAM for a 7B model,” and my RTX 3050 laptop is sitting here calling that a lie. Turns out the model card might be wrong — not the hardware. So AirLLM (25.2k stars, Apache-2.0) runs a full 70B Llama on a single 4GB card, no quantization tricks, and it crossed the Trending list again this week.

AirLLM doesn’t squeeze the model into your VRAM. It streams. The trick is layer-wise inference: instead of loading all 80 layers of a 70B model at once, it pulls one layer into the GPU, computes, discards, and moves to the next.

Peak VRAM stays flat no matter how big the weights get. And that’s why the same package that runs 70B on 4GB also runs Llama 3.1 405B on 8GB and DeepSeek-V3 (671B) on about 12GB. The tradeoff is disk I/O, not memory, which changes the whole game for people stuck with entry-level cards.

AirLLM genuinely runs a 70B model at full precision on 4GB — it never holds more than one layer’s weights in VRAM at a time. There’s no precision being sacrificed to make it fit, and that’s exactly why the accuracy argument holds up. Honestly, it’s a scheduling problem solved on the memory side, not a compression trick.

Quick Start

Install is one line, and inference looks almost identical to standard Hugging Face transformers:

pip install airllm

from airllm import AutoModel

model = AutoModel.from_pretrained("Qwen/Qwen3-32B")

# go bigger with the same call:
# model = AutoModel.from_pretrained("Qwen/Qwen3-235B-A22B")  # 235B, ~3GB
# model = AutoModel.from_pretrained("deepseek-ai/DeepSeek-V3")  # 671B, ~12GB

input_tokens = model.tokenizer(["What is the capital of the United States?"],
    return_tensors="pt", truncation=True, padding=False)

generation_output = model.generate(input_tokens['input_ids'].cuda(),
    max_new_tokens=20, use_cache=True)
print(model.tokenizer.decode(generation_output.sequences[0]))

The AutoModel class auto-detects the model type, so you don’t have to pick between Llama, Qwen, DeepSeek, or Mistral variants — one entry point handles them all. First run decomposes the model and saves it layer-by-layer to your Hugging Face cache, so budget disk space before you start. After that initial split, generation begins almost immediately.

There’s also a compression path: pass compression='4bit' (or '8bit') and AirLLM applies block-wise quantization to the weights only. Because the bottleneck is disk loading rather than compute, it only shrinks the load size instead of quantizing activations — lower accuracy loss than classic quantization, with a claimed 3x speedup.

How It Compares

Here’s where AirLLM sits against the common ways people run big models on weak hardware — if you’re still deciding which model fits your card, my whichllm guide walks the whole selection process:

Quantization attacks the weights to shrink the model; AirLLM attacks the loading pattern. That’s the core difference. You get almost zero accuracy loss because you’re not throwing away precision — you’re just being smart about what’s in memory at any given second. Still, the cost shows up as latency instead.

Hands-On Notes

I ran AirLLM on my Ryzen 7 + RTX 3050 (4GB) laptop with a 32B Qwen model. VRAM hovered around 3.5GB the whole time. What surprised me wasn’t the memory, it was my drive: the model splits across the NVMe, and every layer read is a disk hit — so you want an SSD, not an HDD. Still, a 20-token reply took about 40 seconds. Slow, but it’s a 32B model on a card every other tool refused to even try.

A second surprise came with a sparse MoE model. AirLLM only streams the experts a token actually routes to, which is why Kimi K3 (2.8T parameters — the biggest open-source model released to date) runs in under 4GB of VRAM. I went deeper into running that one on a single box in my Deltafin review. That’s the part that genuinely made me stop and re-read the README.

What to Watch Out For

The honest catch is throughput. Layer streaming trades VRAM for disk bandwidth, so this is a research and tinkering tool, not a production serving layer. If you need low latency per token, AirLLM will frustrate you — expect seconds per token on big models. And verify the compression option’s “3x speedup, almost no accuracy loss” claim against your own eval data before trusting it on anything serious.

Disk footprint is the other gotcha. Decomposing and caching the split model means a 70B download plus layer files can eat 40GB+ of storage. The delete_original=True flag drops the original checkpoint and saves about half that — worth knowing before a big download. Gated models also need a Hugging Face token.

Bottom Line

AirLLM is a genuinely clever reframe of the “your GPU is too small” problem. If you own a low-VRAM card and have been window-shopping 70B models, this is the cheapest way in — free, Apache-2.0, an active maintainer, 25k stars behind it. It won’t replace your API calls for production, but for offline experimentation on models you thought were out of reach, it’s hard to beat. Worth a clone and a long Saturday with your SSD.