I almost scrolled past it, because “runs a frontier model on a potato” is usually marketing. But this one isn’t. FareedKhan-dev’s kimi-k3-in-c hit 607 stars in its first two days — roughly 300 a day, the third-fastest new AI project on GitHub right now — and by the time I checked the API while writing this it was at 633. That pace alone earned a look. The engineering earned the rest of my evening.

TL;DR: the same model, from a cluster to a laptop

Quick verdict up front. kimi-k3-in-c does not quantize, crop, or approximate Kimi K3. Instead it loads the full 1.56TB checkpoint from NVMe and streams the expert weights in and out of memory as it generates. Peak resident set on the laptop preset is 8.24GB. But these are the same weights Moonshot released — the repo’s own docs/data shows the same token IDs emitted byte-identically all the way from an 8GB memory cap up to 224GB. Only the clock changes. The author’s line is worth stealing: “Give it more memory and the answer does not change, only the clock.”

Here’s the thing most people miss. This isn’t “optimized to the point where you don’t need a GPU.” It’s designed so the GPU was never part of the equation. Pure C99, memory-mapped checkpoint, a few labeled pipes between a resident working set on top and the model on NVMe underneath. So compile once, and it runs anywhere Linux x86-64 lives.

The architecture: four reductions instead of a bigger box

The whole project is one question: where do the bytes live? Four decisions take a 16-node cluster down to a laptop.

1. The experts already ship at half a byte. The 1.45TB of routed experts stay packed in their 4-bit form and get multiplied straight out of that — never unpacked to full precision resident in RAM.
2. KDA attention with a memory that never grows. A kernel for the attention block keeps its working set flat instead of scaling with sequence length.
3. MLA collapses 96 heads into one latent. Multi-head latent attention means one latent vector instead of ninety-six separate heads doing the same job.
4. Streaming the trunk turns a floor into a dial. The 93-layer trunk reads layer-by-layer from disk, so the memory ceiling becomes a knob you turn, not a wall you hit.

The dense trunk stays resident to whatever depth you choose; the routed experts are never resident at all. Even so, a cache with a configurable budget holds whichever experts you touched most recently, and the shipped trace tells you how big that cache should be. That’s the “176KB engine, 8.24GB of RAM, 2.78 trillion parameters” trick — and it’s reproducible, because every number is committed with its measurement output.

Though the README is genuinely hostile to hand-waving. It calls out the quantization crowd directly — once you 3-bit re-encode the expert bank, “those weights are no longer the weights Moonshot published.” That’s a real philosophical split, and I’ll get to why both sides are right in the comparison.

Verifying the engine: what I actually did

Let me be precise about the boundary before I quote any number. I cloned the repo, read the README, docs/PERFORMANCE.md, and the raw output in docs/data/, and I pulled the committed test logs in tests/fixtures/gates/. What I could not do is run make test myself — this machine I’m on has no C toolchain and no Linux box handy, and I’m not going to pretend otherwise. So everything below about the test suite is the project’s own committed output, which I verified exists and matches the README’s claims, rather than something I generated.

The honest upside is that you can verify the engine yourself without touching the 1.56TB checkpoint. But even better, you can do it in about a minute.

git clone https://github.com/FareedKhan-dev/kimi-k3-in-c.git
cd kimi-k3-in-c
make -j            # seconds. Seven C files, a compiler, and OpenMP
make test          # under a minute

The suite ends with a verdict line, not a shrug. From the committed gate log:

GATE 1  teacher forcing : 32/32 positions match tf_pred
        generated span  : 20/20  <- must be exact
GATE 2  greedy decode   : 20/20 generated tokens match full_ids
GATE 3  incremental    : 20/20 generated tokens match full_ids  <- KV cache + carried KDA state

VERDICT: ENGINE MATCHES THE REFERENCE EXACTLY

That’s every kernel, the streaming cache, the safetensors reader, the tokenizer, and an end-to-end oracle over a 13-layer model with the same tensor graph as the released one, checked against a PyTorch reference from fixtures committed to the repo. Twenty-two ops tests pass, zero fail. But the point of quoting it: you’re not trusting the README, you’re trusting a test you can run yourself in under a minute.

Requirements are ordinary except for disk. Linux x86-64, a CPU with AVX2 + FMA (AVX-512 optional), 8GB of RAM up, GCC ≥ 9 or Clang ≥ 10. The gate is storage: 1.56TB checkpoint plus a 109GB packed trunk, ideally on fast local NVMe.

Kimi K3 measured: what two presets actually do

The generation numbers below come straight from the README’s own examples and the docs/data/ measurement campaign. They’re the author’s measured results on an AMD EPYC 7763 box (228 GiB RAM, 3.2 TB NVMe at 3.2 GB/s O_DIRECT) — quoted so you can see exactly what “CPU inference on a frontier model” means. So treat them as the project’s numbers, not mine.

Read that second row carefully. 127.92GB of RAM, and you still wait ~11 seconds per token. Yet the answer is byte-identical to what 8GB produced — the only difference is the wall clock. “Only the clock” is doing real work in that sentence.

One honest caveat the project itself insists on: a third of a single measurement is noise. The docs/data/README.md documents a 33% spread across three identical runs of the same binary, same prompt, same flags. Even so, the ladder numbers are reliable as comparisons across memory budgets, not as the engine’s absolute speed. Longer sustained generations (16–32 tokens) actually run faster — 10.66 to 11.79 s/token — because the cold-start pinning cost gets amortized. The 32 s/token laptop figure is the honest worst case, and I’d quote it that way.

So the plain-language version: a single sentence takes a couple of minutes, and a short paragraph is a coffee break. Nobody is running this for daily chat. Yet it earns its place in offline research, privacy-sensitive one-shot inference where the weights never leave your disk, teaching experiments, and “I want to prove a 2.78T model can run on the box I already own.” For interactive use, this is not it, and I’ll say that plainly.

The trade-off: quantization vs. full-fidelity streaming

kimi-k3-in-c sits at one end of a real design spectrum, and it’s worth seeing where. It’s also the natural sibling to a project I covered last week — deltafin asks how fast frontier inference can go on hardware you own; kimi-k3-in-c asks how little hardware it can run on. Same Kimi K3, opposite philosophies.

Each row is a different trade. Still, llama.cpp gets you speed through quantization, but those weights aren’t the ones Moonshot shipped anymore. kimi-k3-in-c keeps every byte and pays for it in clock time. Deltafin goes the other way and wants a workstation. And a cloud API is instant and costs a subscription, but your prompt leaves your machine. So there’s no wrong answer here — just a “what are you optimizing for” answer.

My take: if model fidelity or data sovereignty is the whole point, the streaming route wins. If you need tokens back in seconds, you don’t want this. Different trade-offs, and this project is honest that it picked the slow lane.

Can you run Kimi K3 on a VPS?

This is where the project turns into an infrastructure question, because the two presets map cleanly onto two price tiers.

The laptop preset wants 8.24GB of RAM and roughly 1.7TB of disk. But that’s reachable on an 8GB or 16GB droplet for a fraction of what a GPU box costs — and it means the full-fidelity model, not a quantized imitation, running on hardware that isn’t pretending to be a data center. While the server preset, at 127.92GB, is a memory-optimized instance problem. Here DigitalOcean’s memory-optimized droplets scale up into that range, which makes them the natural fit for the full-budget config.

# On a memory-optimized VPS, once the checkpoint lives on fast NVMe:
git clone https://github.com/FareedKhan-dev/kimi-k3-in-c.git
cd kimi-k3-in-c && make -j
./bin/k3 ~/k3model --trunk ~/k3trunk --preset server \
    --tok ~/k3model --prompt "def fibonacci(n):" --gen 28 --incremental

That’s the honest framing the project earns: you don’t need a GPU to run a frontier model — you need a box with RAM and disk, and the right preset. A memory-optimized VPS from DigitalOcean gives you the RAM tier the server preset needs; Vultr’s high-frequency and large-memory lines are a solid regional alternative; and if you’re on the 8GB laptop path, Hostinger’s budget tier gets you there cheapest. But the 1.56TB checkpoint is the real cost, and it wants a fast 2TB NVMe drive — one of the well-reviewed options from Amazon’s storage aisle (affiliate link — ASIN pending verification, see publisher note).

Another boundary note so I’m not overselling: I verified the code and committed data on my side, and I’m quoting the project’s measured numbers for generation. I did not run a full 1.56TB inference on a rented VPS, so treat those VPS timing expectations as “should run” rather than “I measured this here.” If you actually pull down the checkpoint, your mileage will vary at 32 seconds per token — probably by a lot depending on your NVMe and how patient you are.

The model weights are a separate question

The code is Apache-2.0, no argument there. But the model itself is Moonshot’s Kimi K3, distributed on HuggingFace under its own license — and that’s a separate thing from the engine that runs it. If you’re going to use this beyond a personal experiment, check the weights’ license yourself before you deploy anything that touches a business. I’m flagging it because “the repo is Apache-2.0” is easy to read as “everything here is free to ship,” and the weights are a distinct layer with their own terms. That’s not legal advice; it’s me telling you to look before you build on it.

Who should actually use this

• The no-GPU developer who wants to run a genuine 2.78T model, not a quantized stand-in, on the laptop they already own.
• The self-hoster / privacy-first user whose whole point is that weights never leave the machine.
• The engineer curious about MoE inference — the README is one of the clearest written explanations of expert streaming and attention memory I’ve read in this space.
• Not someone who wants conversational speed. 32 s/token is a feature of the trade-off, not a bug, and anyone selling you this as fast is lying to you.

If you want the speed side of that coin, a model server like deltafin or a quantized stack is the better tool — different goals. If you want token-cost optimization instead of token-time optimization, my headroom review is the other end of that spectrum, and if you’re curious about the fine-tuning side, my llamafactory quick review pairs with this nicely.

The bottom line

Here’s my verdict. kimi-k3-in-c is the most honest piece of local-inference engineering I’ve seen this month, precisely because it doesn’t pretend to be fast. It’s a proof that “frontier model” and “8GB of RAM” can coexist — not through clever lossy compression, but through ruthless decisions about where every byte lives. The star count went from 607 to 633 while I was writing this, and that tells me I’m not the only one who found it.

Run the one-minute test suite first. If the engineering hooks you the way it hooked me, a memory-optimized VPS is genuinely enough to run the real thing — full weights, no GPU, no cloud round-trip. That’s not hype. That’s what 176KB of C99 buys you.

Disclosure: Some links below are affiliate links. If you sign up through them, I may earn a commission at no extra cost to you.

• Vultr — high-memory cloud GPU instances from $6/mo
• DigitalOcean — $200 credit for new users, great for GPU droplets

So Kimi K3 dropped two days ago and instantly became the most exciting open-weight model release this month — 7,300+ stars, 2.78 trillion parameters in a Mixture-of-Experts architecture, multimodal. Problem: nobody has a cluster sitting around to serve it. Enter Deltafin, a research project that does exactly what its README says: “runs a Mixture-of-Experts model far larger than the machine it sits on.” I spent the evening digging through its codebase, and honestly? The ambition matches the execution.

What Deltafin actually does with Kimi K3

Deltafin is a Python-native runtime that loads Kimi K3 onto a single Mac or Linux workstation and exposes it through an OpenAI-compatible API. Every token routes through 16 active experts × 92 layers — that’s 25.8 GB of expert data per token, which on the developer’s reference M1 Max (64 GB) clocks in at a measured 0.0687 tokens/second (about 14.6 seconds per token). That is not a typo. It’s slow. And it’s also completely unprecedented for a model this size on consumer hardware.

Plus, the whole thing supports three backends: MPS on Apple Silicon, CUDA on NVIDIA, and a pure CPU fallback. Expert weights use native MXFP4 precision — a full custom kernel pipeline (the tools/fused_gemv.c file alone is ~45KB of C code). And the build system auto-detects your platform, probes for NVCC, and compiles the right native libraries for your exact GPU architecture. If your local machine doesn’t have the RAM or disk, spinning up a high-memory GPU instance on Vultr (affiliate link) or DigitalOcean lets you run Deltafin without the hardware investment. If you’ve used other local LLM tools like LlamaFactory for fine-tuning, the engineering depth here is on a different level entirely.

Two modes, one trade-off — Deltafin’s local Kimi K3 tradeoffs

In practice, full mode downloads all 1.7 TB of expert weights to local disk. Sure, streaming keeps only a sparse cache — it fetches missing experts on demand from Hugging Face. The README is refreshingly honest about this — streaming mode at inference means you’re effectively bottlenecked by your internet connection, which can stretch a single chat completion into “hours” territory.

I spent a while reading the tools/serve_openai.py server implementation. It’s about 300 lines, and the design notes include gems like: “Concurrency would be meaningless at this speed” and “Set your client’s timeouts to hours, not seconds.” That’s the tone of a project that knows exactly what it is.

Running Deltafin: OpenAI server setup for Kimi K3

python tools/serve_openai.py --port 8000

That’s it. On a remote VPS with enough RAM, this works identically — deploy to a Vultr or DigitalOcean high-memory instance and your OpenAI SDK client can talk to K3 from anywhere. Then any tool that reads OPENAI_BASE_URL can talk to Kimi K3:

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="none")
r = client.chat.completions.create(
    model="deltafin-kimi-k3",
    messages=[{"role": "user", "content": "Explain MLA attention in three sentences."}]
)
print(r.choices[0].message.reasoning_content)  # K3's thinking trace
print(r.choices[0].message.content)            # the answer

So the server implements /v1/chat/completions, /v1/completions, and /v1/models. And streaming works ("stream": true). K3’s thinking section is split into reasoning_content and content — DeepSeek-style — so you see the model’s chain of thought before the answer. A deterministic response memo caches identical requests (last 32), which is smart given the generation speed.

One detail I noticed: the server holds a global threading.Lock(). No concurrent generations. The README acknowledges this and doesn’t apologize — it’s greedy-only, temperature/top_p are accepted and silently ignored. This isn’t a production inference server. It’s a research tool for poking at a 2.78T model from your desk.

Who should actually run Deltafin’s Kimi K3 locally

| Look, Deltafin is not for everyone. The full install needs 1.7 TB of free disk and a willingness to wait 5–10 hours for the download. And the streaming install is faster to set up but slower to use. On both paths you’re getting roughly 4 tokens per minute on the developer’s reference hardware — a 2019-era M1 Max. If you don’t have a workstation that can handle the load, renting a cloud instance from DigitalOcean or Vultr is a straightforward alternative — you pay only for the hours you use.

But here’s the thing: no other open-source project can run a 2.78T MoE model on a single workstation. Deltafin does it with custom MXFP4 kernels, a well-structured build system, and an API that works with any OpenAI SDK client. And the codebase (88 Python tools, C/CUDA/Metal kernels, a 37K-line build_native.py) shows serious engineering. If you’re exploring the local LLM space, I’ve also covered tools like Headroom for cutting agent token costs — but Deltafin is in a category of its own.

My verdict: If you have the disk space and the patience, Deltafin is the most interesting local LLM project I’ve seen this month. But if you don’t have 1.7 TB lying around, wait for the quantized versions — bookmark this repo. And when the hardware catches up, this architecture will be the blueprint.

Deltafin is a research project, not a production service. Some links above are affiliate links — if you sign up through them, I earn a commission at no extra cost to you. As an Amazon Associate I earn from qualifying purchases.