Disclosure: Some links below are affiliate links. If you sign up through them, I may earn a commission at no extra cost to you.

• Vultr — high-memory cloud GPU instances from $6/mo
• DigitalOcean — $200 credit for new users, great for GPU droplets

So Kimi K3 dropped two days ago and instantly became the most exciting open-weight model release this month — 7,300+ stars, 2.78 trillion parameters in a Mixture-of-Experts architecture, multimodal. Problem: nobody has a cluster sitting around to serve it. Enter Deltafin, a research project that does exactly what its README says: “runs a Mixture-of-Experts model far larger than the machine it sits on.” I spent the evening digging through its codebase, and honestly? The ambition matches the execution.

What Deltafin actually does with Kimi K3

Deltafin is a Python-native runtime that loads Kimi K3 onto a single Mac or Linux workstation and exposes it through an OpenAI-compatible API. Every token routes through 16 active experts × 92 layers — that’s 25.8 GB of expert data per token, which on the developer’s reference M1 Max (64 GB) clocks in at a measured 0.0687 tokens/second (about 14.6 seconds per token). That is not a typo. It’s slow. And it’s also completely unprecedented for a model this size on consumer hardware.

Plus, the whole thing supports three backends: MPS on Apple Silicon, CUDA on NVIDIA, and a pure CPU fallback. Expert weights use native MXFP4 precision — a full custom kernel pipeline (the tools/fused_gemv.c file alone is ~45KB of C code). And the build system auto-detects your platform, probes for NVCC, and compiles the right native libraries for your exact GPU architecture. If your local machine doesn’t have the RAM or disk, spinning up a high-memory GPU instance on Vultr (affiliate link) or DigitalOcean lets you run Deltafin without the hardware investment. If you’ve used other local LLM tools like LlamaFactory for fine-tuning, the engineering depth here is on a different level entirely.

Two modes, one trade-off — Deltafin’s local Kimi K3 tradeoffs

In practice, full mode downloads all 1.7 TB of expert weights to local disk. Sure, streaming keeps only a sparse cache — it fetches missing experts on demand from Hugging Face. The README is refreshingly honest about this — streaming mode at inference means you’re effectively bottlenecked by your internet connection, which can stretch a single chat completion into “hours” territory.

I spent a while reading the tools/serve_openai.py server implementation. It’s about 300 lines, and the design notes include gems like: “Concurrency would be meaningless at this speed” and “Set your client’s timeouts to hours, not seconds.” That’s the tone of a project that knows exactly what it is.

Running Deltafin: OpenAI server setup for Kimi K3

python tools/serve_openai.py --port 8000

That’s it. On a remote VPS with enough RAM, this works identically — deploy to a Vultr or DigitalOcean high-memory instance and your OpenAI SDK client can talk to K3 from anywhere. Then any tool that reads OPENAI_BASE_URL can talk to Kimi K3:

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="none")
r = client.chat.completions.create(
    model="deltafin-kimi-k3",
    messages=[{"role": "user", "content": "Explain MLA attention in three sentences."}]
)
print(r.choices[0].message.reasoning_content)  # K3's thinking trace
print(r.choices[0].message.content)            # the answer

So the server implements /v1/chat/completions, /v1/completions, and /v1/models. And streaming works ("stream": true). K3’s thinking section is split into reasoning_content and content — DeepSeek-style — so you see the model’s chain of thought before the answer. A deterministic response memo caches identical requests (last 32), which is smart given the generation speed.

One detail I noticed: the server holds a global threading.Lock(). No concurrent generations. The README acknowledges this and doesn’t apologize — it’s greedy-only, temperature/top_p are accepted and silently ignored. This isn’t a production inference server. It’s a research tool for poking at a 2.78T model from your desk.

Who should actually run Deltafin’s Kimi K3 locally

| Look, Deltafin is not for everyone. The full install needs 1.7 TB of free disk and a willingness to wait 5–10 hours for the download. And the streaming install is faster to set up but slower to use. On both paths you’re getting roughly 4 tokens per minute on the developer’s reference hardware — a 2019-era M1 Max. If you don’t have a workstation that can handle the load, renting a cloud instance from DigitalOcean or Vultr is a straightforward alternative — you pay only for the hours you use.

But here’s the thing: no other open-source project can run a 2.78T MoE model on a single workstation. Deltafin does it with custom MXFP4 kernels, a well-structured build system, and an API that works with any OpenAI SDK client. And the codebase (88 Python tools, C/CUDA/Metal kernels, a 37K-line build_native.py) shows serious engineering. If you’re exploring the local LLM space, I’ve also covered tools like Headroom for cutting agent token costs — but Deltafin is in a category of its own.

My verdict: If you have the disk space and the patience, Deltafin is the most interesting local LLM project I’ve seen this month. But if you don’t have 1.7 TB lying around, wait for the quantized versions — bookmark this repo. And when the hardware catches up, this architecture will be the blueprint.

Deltafin is a research project, not a production service. Some links above are affiliate links — if you sign up through them, I earn a commission at no extra cost to you. As an Amazon Associate I earn from qualifying purchases.