Running a 284-billion-parameter language model on a laptop sounded absurd two years ago. It works now, and it’s starting to reshape the economics of AI infrastructure.
No single invention caused this. It’s the combination of three: Mixture-of-Experts models that only activate a fraction of their parameters per token, quantization that compresses models down to a fraction of their original size, and an inference engine, DwarfStar (DS4), built to run that stack well on consumer hardware.
The whole point is to pull AI out of the cloud and run it on machines you own.
The model: DeepSeek V4 Flash
DeepSeek V4 Flash is a Mixture-of-Experts language model with 284 billion total parameters, but only about 13 billion are active for any given token.
That gap between total and active parameters is why MoE models punch above their weight. Every parameter must fit in memory, but only the activated experts get computed. That’s how V4 Flash delivers frontier-grade reasoning and instruction following while using far less compute than a dense model of similar size.
The model ships in two variants: Flash, tuned for efficiency, and PRO, which trades memory for higher capability. Both use Multi-head Latent Attention (MLA) to compress the KV cache, cutting it by up to 93% versus standard attention. That compression is what makes long-context inference practical; V4 Flash handles up to 1 million tokens of context.
It’s MIT licensed, so you can run, modify, and deploy it without permission. The weights are open, and that matters for anyone who wants to host their own reasoning models.
The engine: DwarfStar (DS4)
Enter DwarfStar, or DS4: a native inference engine from Salvatore Sanfilippo, better known as antirez, the creator of Redis. Released in May 2026, it currently sits at 17.5k GitHub stars.
DS4 is deliberately narrow. It’s not a general-purpose GGUF runner like llama.cpp, and it’s not a wrapper around another runtime. It’s a self-contained, pure-C engine optimized specifically for DeepSeek V4 Flash and PRO. The logic: new models ship constantly, and generalist runtimes end up chasing whichever one is newest. DS4 instead focuses on one model at a time and does it properly.
The project is developed with strong assistance from GPT 5.5, which antirez says openly: “If you are not happy with AI-developed code, this software is not for you.”
What makes DS4 different
DS4 handles the whole pipeline itself: model loading, prompt rendering, tool calling, KV state management in RAM or on disk, a server API, and even a built-in coding agent. It’s less a model runner and more a finished product around the model.
The engine supports three backends: Metal on macOS, the primary target tuned for Apple Silicon; CUDA on Linux, with special attention to NVIDIA’s DGX Spark; and ROCm on Linux for AMD Strix Halo systems.
There’s also a CPU path, but antirez notes a macOS bug that causes kernel crashes with CPU inference: “each time you have to restart the computer, which is not funny.”
The KV cache is a disk citizen now
The most interesting part of DS4 might be how it treats the KV cache. The project’s assumption: with compressed KV caches and fast SSDs, the cache doesn’t have to live in RAM. On a modern MacBook, the KV cache is a first-class disk citizen, which turns RAM from a hard cutoff (can I run this model?) into a spectrum of speed.
Quantization: the 7:1 compression
DeepSeek V4 Flash in FP16 is roughly 568 GB, far beyond any consumer machine. DS4 handles it with selective, mixed-precision quantization.
Standard quantization applies the same bit reduction uniformly across all layers. DS4 takes a different approach: not all layers contribute equally to model quality. High-sensitivity layers (early attention, output-proximal layers, specific MLP components) are kept at higher precision, while many intermediate layers tolerate much heavier compression.
The result for V4 Flash:
| Format |
Size |
Best for |
| Q4-IMatrix |
~256 GB |
256GB+ RAM machines |
| Q2-Q4-IMatrix |
~96 GB |
96-128 GB RAM machines |
| Q2-IMatrix |
~81 GB |
96-128 GB RAM machines |
| PRO Q2 |
~256 GB |
512 GB RAM machines |
The 2-bit quantization is surprisingly usable under coding agents and tool calling. It’s asymmetrical: only the routed MoE experts drop to IQ2_XXS / Q2_K, while shared experts, projections, and routing layers stay at higher precision. That asymmetry is where the quality preservation comes from.
Hardware requirements: what you actually need
The barrier to entry has dropped a lot. Here’s the hardware picture in 2026:
MacBook Pro M3/M4 Max (128 GB unified memory)
A 128 GB MacBook Pro M3 or M4 Max is the sweet spot. The ~81 GB Q2-IMatrix build leaves about 47 GB for the OS, KV cache, and overhead, and DS4 hits roughly 26-27 tokens/second on a short prompt, or 21-25 tokens/second with long contexts (11k+ tokens).
The M5 Max reaches 34 tokens/second on short prompts.
MacBook Pro M3/M4 Ultra (192-512 GB unified memory)
On the Ultra-class chips the Q4 build becomes viable. A Mac Studio M3 Ultra with 512 GB runs Q4 at 35-37 tokens/second, and even PRO Q2 at roughly 9.5 tokens/second, which is slow but usable for inspection and occasional work.
NVIDIA DGX Spark (128 GB unified memory)
NVIDIA’s compact personal AI supercomputer, built around the GB10 Grace Blackwell Superchip, hits around 13-14 tokens/second with V4 Flash Q2. The Blackwell GPU does tensor math faster than Apple Silicon, and native CUDA means broader framework compatibility.
AMD Strix Halo
AMD’s Strix Halo platform (found in systems like the Framework Desktop) uses unified memory like Apple Silicon. DS4 supports ROCm for these, though benchmarks are still emerging.
SSD streaming: running models larger than RAM
DS4 also has an SSD streaming mode for models bigger than your RAM. The non-routed weights stay resident in memory, while routed MoE experts live in an in-memory cache and load from the GGUF file on cache misses.
Modern Mac SSDs are fast enough to make cache misses tolerable. Long prefills can still be fast; generation is more sensitive to cache misses because every new token routes through experts again.
On a 64 GB MacBook, you can run the Q2 Flash GGUF with SSD streaming and a 32 GB expert cache:
./download_model.sh q2-imatrix
./ds4 -m ./ds4flash.gguf \
--ssd-streaming \
--ssd-streaming-cache-experts 32GB \
--ctx 32768 \
--nothink
This turns the question from “can I run this model?” into “how fast will it run?”
Distributed inference: combining multiple machines
For the truly ambitious, DS4 can split inference across multiple machines. Run the full PRO Q4 build across two 512 GB Mac Studios by splitting transformer layers: one machine handles layers 0-30, the other takes layers 31 through output.
Prefill is pipelined: on two M5 Max machines linked by Thunderbolt 5, a 63k-token prompt saw a 1.85x prefill speedup. Generation stays strictly autoregressive, so distributed generation is actually slower than single-machine because of cross-machine activation hops per token. The win is capacity, not speed.
A more accessible alternative: Qwen3.6-35B-A3B
Not everyone has 96 GB of RAM. If you’re on more modest hardware, Qwen3.6-35B-A3B is worth a look.
Released in April 2026 under Apache 2.0, it’s a 35B-parameter MoE model that activates only 3B per token, with a hybrid of Gated DeltaNet, MoE, and Gated Attention. For its size it’s genuinely impressive.
SWE-bench Verified: 75.0%, ahead of Qwen3.5-27B and Gemma4-31B.
Hardware requirements for Qwen3.6-35B-A3B
| Precision |
VRAM/RAM Needed |
Hardware |
| Q4_K_M |
~18-20 GB |
24GB GPU (RTX 3090/4090) |
| IQ4_XS |
~14-16 GB |
16GB GPU + KV cache optimization |
| Q8_K_XL |
~36 GB |
48GB GPU or CPU+GPU hybrid |
| CPU-only |
~70 GB |
64-128GB RAM desktop |
It runs comfortably on a 24 GB RTX 3090/4090 at Q4. On CPU-only boxes with lots of RAM (64-128 GB DDR5) it works, just slower, and a Mac Studio or Max with 64-128 GB unified memory handles it well.
It’s on Ollama (ollama run qwen3.6:35b-a3b), in GGUF for llama.cpp, and serveable via vLLM. Community quantized builds trade off quality differently; the IQ4_XS version by oamazonasgabriel is tuned for 24 GB VRAM.
DS4 is one option in a busy local inference scene. The short version:
llama.cpp is the foundation. 119k GitHub stars, plain C/C++ with no dependencies, quantization from 1.5-bit to 8-bit, and it runs on everything from Apple Silicon to RISC-V. DS4 exists because llama.cpp and GGML opened that path.
Ollama is the easiest on-ramp: one command, model management, and an API server, with GGUF support through the llama.cpp backend. Good for getting started fast.
vLLM does high-throughput serving for production with PagedAttention, continuous batching, and tensor parallelism. Reach for it when you need to serve many concurrent requests.
Unsloth is fast fine-tuning and inference with a web UI, GGUF export, 2x faster training, and 70% less VRAM.
Pick by what you’re doing: llama.cpp for maximum compatibility, Ollama for simplicity, vLLM for production, DS4 for DeepSeek-specific optimization.
Why run AI locally?
The reasons are practical:
Cost. API pricing for frontier models adds up fast. A single long-context reasoning session can cost more than the hardware to run locally.
Privacy. Your data stays on your device, with no calls to external APIs. That matters for anyone handling confidential information.
Latency. No round-trip to a cloud server, which matters for interactive applications.
Control. No rate limits, no provider changing the model under you, and you own the deployment.
Offline. Works without internet, which helps for travel, secure environments, or plain reliability.
The tradeoffs
There are real costs too:
Hardware. DS4’s Flash model needs at least 96 GB of RAM, which means a MacBook Pro M-series at the top memory tier or a Mac Studio. Not exactly budget hardware.
Quality. Quantization costs you something. The selective approach keeps most capability, but you’ll notice degradation in very long-context retrieval, exact numerical reasoning, and some creative writing nuance.
Lock-in. DS4 only runs the GGUF files published for this project; it’s not a general loader. If a better open-weight model appears, DS4 may switch or drop support. As antirez puts it, “the project is strictly opportunistic.”
SSD wear. Constant model loading and streaming wears down your SSD. For most users it’s a fair trade, but it’s worth knowing.
Getting started
A practical path if you want to try local inference today:
-
Start small. Run Qwen3-8B or DeepSeek-R1 8B on any modern GPU or even CPU, and get comfortable with the tooling.
-
Move to mid-range. An RTX 3090/4090 (24 GB) or an M-series Mac with 32-48 GB runs 32B-class models comfortably at Q4.
-
Go big. 96-128 GB of unified memory opens the door to DeepSeek V4 Flash via DS4, where frontier capability meets local execution.
-
Experiment with quantization. Try different levels and measure the quality tradeoffs for your use case; the right balance depends on your workflow.
The DS4 project itself is straightforward to set up:
git clone https://github.com/antirez/ds4
cd ds4
make # macOS Metal
# or
make cuda-generic # Linux CUDA
./download_model.sh q2-imatrix
./ds4
The project is beta quality, and antirez is upfront about it. It’s usable and under active development, and it’s one of the more ambitious local-inference efforts I’ve seen.
What this means for the industry
Running frontier-class models locally changes the economics. Open weights, a purpose-built inference engine, and consumer hardware mean the cloud stops being a requirement and becomes a choice.
The MoE shift is accelerating this. DeepSeek V4 Flash (284B total, 13B active), Qwen3.6-35B-A3B (35B total, 3B active), and gpt-oss (20B on consumer GPUs) show that total parameter count no longer sets your hardware requirement, because only active experts get computed during inference.
Better quantization (Unsloth Dynamic 2.0, IQ4_XS, APEX) gets near-lossless quality at 4-bit. Apple Silicon’s unified memory runs 70B+ models on one machine, and llama.cpp improvements make large models feasible on RAM-only systems.
“Local first” is becoming a practical strategy rather than a privacy ideal. The hardware, models, and inference engines are all here. The choice now is whether you want to run AI yourself, not whether you can.