For years, “running AI locally” meant wrestling with CUDA drivers and hoping your NVIDIA GPU had enough VRAM.
That’s changing fast. Over the last three months, Apple Silicon has gone from “nice to have” to a great choice for local AI development. And Ollama’s MLX engine is the reason.
The timeline
The story starts in March. Ollama shipped a preview of MLX-powered inference on Apple Silicon, built on Apple’s own MLX framework to leverage the unified memory architecture. On an M5 chip, prefill jumped from 1154 to 1810 tokens/s and decode from 58 to 112 tokens/s using Qwen3.5-35B-A3B with NVFP4 quantization, compared to Q4_K_M quantization in Ollama 0.18. (Ollama blog, March 30, 2026)
In June, Ollama shipped a major MLX engine update with NVFP4 support, which cuts the quality loss of 4-bit quantization roughly in half compared to the traditional q4_K_M format, and adds about 20% faster output. (Ollama blog, June 11, 2026)
Then came multi-token prediction (MTP) for Gemma 4 on June 29, which generates tokens nearly 90% faster on the Aider polyglot benchmark. The benchmark isn’t synthetic; it was measured by running a real coding agent through real programming tasks.
Why it matters for developers
If you’ve ever used a local coding agent, you know what it actually means: less waiting between tool calls, faster file reads, and agents that feel responsive instead of sluggish.
Multi-token prediction, done right
MTP works by having a small draft model propose several tokens at once, which the main model then verifies in a single pass. Code is especially predictable, full of closing brackets, repeated identifiers, and boilerplate, so the draft model’s proposals get accepted often. This matters most for coding agents, which call the model continuously as they read files, run tools, and work through tasks. (Ollama blog, June 29, 2026)
The hard part isn’t the draft-verify loop itself. It’s keeping the loop reliable, because the ideal number of tokens to draft changes moment to moment, and drafting too many can make MTP slower than not speculating at all. Ollama solves this by auto-tuning the draft length at runtime: it tracks acceptance rates and verification times, selecting the length that produces the most tokens per second, and adjusts as the text changes. When proposals stop being accepted, it falls back to plain one-at-a-time decoding, so speculation never slows generation down. (Ollama blog, June 29, 2026)
Each verification round runs entirely on the GPU as a single pass (drafting, sampling, verification, and post-verification sampling) with no return to the CPU. Accepted tokens are kept. Rejected ones are handled by recording a rollback point before each proposal, so a rejection only rewinds to the last accepted token without touching anything earlier. (Ollama blog, June 29, 2026)
Ollama also contributed a custom MLX kernel for this workload, a matrix multiplication kernel optimized for the awkward batch sizes (2–8 tokens) that speculative decoding produces. Matrix multiplication kernels are typically built for either a single token (decode) or a large batch (prefill), and a handful of draft tokens falls between the two. The kernel reads and unpacks each block of weights once and reuses it across the entire batch, rather than re-reading the weights for every token. On an M5 Max with nvfp4, this makes Gemma 4’s largest matrix multiplications 2× to 2.5× faster. The kernel is available for other models to use too. (Ollama blog, June 29, 2026)
Agent-aware caching
A coding agent’s workload is dominated by prompt processing. Every tool call is a new request, and every request resends the whole transcript: system prompt, tool definitions, and every file read so far. Over a single task the model ends up processing the same context dozens of times.
Ollama’s new snapshot system saves model state at key points across conversations, using the same approach that serves agent workloads in Ollama’s cloud:
- With multiple agents, one hands off to a subagent and picks back up later, or two sessions run at the same time. Each one resumes from its own saved state, and anything they have in common (often tens of thousands of tokens of system prompt, tool definitions, and ingested files) is only processed once.
- With thinking models, reasoning tokens are generated, then dropped from the conversation history, so the next request never matches the state the engine just built. A snapshot taken right before the response starts gives the next turn somewhere to resume from.
- With branching and retries, a different follow-up or a regenerated response diverges from the cached conversation instead of extending it. Because snapshots exist where conversations split, only the new direction needs to be processed.
Most new models make this harder than it sounds. Sliding-window attention and recurrent layers carry state that can’t be rewound. Once the model moves past a point in the conversation, that point can’t be recovered unless state was saved at the time. Ollama saves state at the points conversations are likely to return to: where they branch, at intervals through long prompts, and just before each response. Keeping snapshots selective and incremental leaves more memory for the model. (Ollama blog, June 11, 2026)
It’s not just Apple
Ollama 0.30 added GGUF support through llama.cpp in June, bringing model compatibility to a wider range of hardware including AMD and Intel GPUs via Vulkan. NVIDIA hardware saw up to 20% faster throughput. It doesn’t erase Apple Silicon’s lead for local dev, where the MLX optimizations are still the most mature, but it means Ollama is no longer Apple-only.
The ecosystem signal
Ollama just announced it raised $88M and now serves 8.9 million developers, with 85% of the Fortune 500 using it. (Ollama blog, July 9, 2026) The funding round included Peter Fenton at Benchmark, Tomasz Tunguz at Theory Ventures, Alex Kolicich at 8VC, Y Combinator, Garage Capital, Pace Capital, and angel investors including Solomon Hykes (founder of Docker), Aaron Katz (CEO of ClickHouse), and Spencer Kimball (co-founder of Cockroach Labs).
More users means more investment in platform-specific optimizations, which means better performance for everyone. Apple Silicon’s MLX engine is the clearest example of that cycle.
What to try
If you’re on a Mac and haven’t tested the current state of local inference, here’s where to start (requires Ollama 0.31 or later):
ollama run gemma4:12b-mlx — the fastest model on Apple Silicon right now, with MTP enabled by defaultollama launch pi --model gemma4:12b-mlx — run it through Pi agent- If you want something closer to your current setup:
ollama run qwen3.5:35b-a3b (or the NVFP4 variant)
The gap between cloud and local keeps shrinking. Apple Silicon and MLX are a big part of why.
Sources