I've been running Strata for the better part of a week. The performance made me positively surprised: in the best case scenario, upwards of 47.4 tokens/s of generation and 600 tokens/s prefill. In particular, slow prefill speeds was causing me headaches all the way back when I used DeepSeek V4 Flash, at around 200 tokens/s. While not as good as what you'd see on a DGX Spark or a dedicated GPU, I'll take my wins where I can get.
Strata is a free, MIT-licensed inference engine (C++/CUDA/HIP/SYCL, with its own Metal backend for Apple Silicon) built on parts of llama.cpp / ggml. It runs Qwen3.8-Flash-Next, plus the Coder, Swift 1.5, and Unsloth variants, on one ordinary machine. The front end is a localhost API that speaks both the OpenAI and Anthropic formats, a web app with a live hardware monitor, and an MCP server, so Claude Code, Codex, and friends connect like any hosted provider.
How Strata works
The model is the same one I wrote about in the Qwen3.8-Flash-Next preview: 125B parameters, but only about 6B active per token, because 24,576 tiny experts take turns and 10 answer each call. Strata works that asymmetry from several directions:
- Tiered expert residency. The GPU keeps the most-used experts in an adaptive cache (ranked by an expert profile built from real prompt traces, following the conversation as you chat). All experts sit pinned in system RAM. The CPU computes the experts that aren't on the GPU at the same time the GPU works on the ones that are. The SSD holds the n-gram table and catches the overflow.
- Speculative decoding with the model's own MTP head. No draft model: Qwen's trained-in multi-token-prediction layer guesses up to 3 tokens, and one pass over the 48 layers verifies them, for a 1.6-1.8x speedup with identical output. A prompt-lookup n-gram drafter adds free speed on repeated text.
- Chunked prefill with overlap. Prompts are read in chunks of up to 8,192 tokens, with the next layer's experts streaming while the current layer's attention runs. The measured payoff on a gaming PC is over 1,000 tokens/s of prompt read, about 2,650 tokens/s at 32K on an RTX 5070.
- Quantization as memory budget. Every byte saved on KV cache or weights converts into more GPU-resident experts, which converts into speed. Strata ships 4-bit Hadamard-rotated KV caches and leans on Unsloth's dynamic quants for the quality tier.
- OS-level tricks. Pinned memory, page-cache reuse of mmap'd expert files,
posix_fadviseread-ahead tuned for NVMe, and elastic VRAM segments that lend memory to other programs and take it back.
Strata on the Mac
There is no PCIe link to pipeline over and no separate VRAM: the whole 128 GB is one pool. Strata's Metal backend maps the model file and keeps it resident, reads the n-gram table from the SSD unbuffered (it deliberately doesn't count against the memory budget), and for packs too big to hold resident it streams experts from the SSD into a pinned arena. UD-Q4_K_XL needs about 91 GB, which is why setup recommends it exactly at 128 GB and drops to smaller quants below.
Install is git clone plus ./setup.sh: it checks the chip and free disk, builds the engine from source on your Mac (about ten minutes; there is no prebuilt Mac engine in the releases), downloads the model, and starts the server on http://127.0.0.1:8080. I left the defaults alone, which on Mac means MTP speculative decoding on (--spec 2).
The numbers
Decode speed is tokens/s generating on ordinary chat. Repo rows are greedy at the setup defaults, from docs/MAC.md and the README; my row is five runs through the API at the server's default sampling.
| Machine | Quant | Decode | Notes |
|---|---|---|---|
| M1 Ultra Mac Studio, 128 GB | UD-Q4_K_XL | 47.4 tok/s | my five-run bench, median 47.5 |
| M5 Max, 128 GB | UD-Q4_K_XL | 94.0 tok/s (60.7 without MTP) | repo-measured flagship |
| M5 Max, 128 GB | UD-Q4_K_XL | 48.4 tok/s | llama.cpp Metal, same file |
| M4 Mac mini, 32 GB | Q2_0 | 14.1 tok/s | experts streamed from SSD |
| RTX 5070 12 GB + Ryzen 5 7600 | Q2_0 | 94 tok/s | the PC headline number |
Two things stand out. First, on the same M5 Max the engine runs 1.5x faster than llama.cpp Metal on the same 4-bit file, which is the MTP head and the overlap doing the work. Second, my M1 Ultra sits four chip generations behind that M5 Max and still lands at 47.4 tok/s on a 4-bit quant, against the 37 tokens/s I measured in September. About 28% faster and a higher-quality quant, I definitely can't complain about that win.
The caveats: the first prompt after a cold start is slower while macOS reads the model file, batched parallel requests aren't ported to Metal yet, and the Mac docs mark a few features (hosted-style tuning, some context sizes) as untested on Apple Silicon. None of it bit me in a week of normal chat and coding-agent use.
The good news
I keep getting performance upgrades with a comparable quality to when I started running DeepSeek V4 Flash (17 tokens/s decode, 200 tokens/s prefill). Strata also opens running this model on out-of-the-box gaming PCs, which is a very welcome change of pace. If you own a 64 GB+ Apple Silicon Mac or a 12 GB discrete GPU with 32/64 GB of RAM, give it a try.