Why Your M-Series Mac Can't Keep Up: The Reality of Local AI Inference on Apple Silicon

If you've recently picked up a MacBook Pro with M4 Max and 128GB of unified memory, you probably imagined yourself running a capable local AI coding assistant without breaking a sweat. After all, 128GB of unified memory should mean serious muscle for running large language models, right?

Well, as one developer discovered the hard way, the reality is considerably more complicated. Their journey through custom Metal inference engines, aggressive optimization passes, and multiple model architectures ultimately revealed something frustrating: there's a ceiling around 80-150 tokens per second once you're running anything close to useful model sizes.

The Hardware Promise vs. Reality

Apple Silicon brought genuine innovation to the table with unified memory architecture. No more shuffling data between CPU and GPU memory—everything lives together, theoretically enabling faster processing for AI workloads. And for many tasks, this holds true.

But when it comes to serving large language models for coding assistance, the numbers tell a different story. Even with aggressively optimized inference engines built specifically for Metal, performance tends to plateau at predictable levels:

• Llama.cpp Q4_0: Around 70.9 tokens/second
• MLX 4-bit: Approximately 80.6 tokens/second
• Custom optimized Qwen3-Coder-Next: Roughly 120 tokens/second
• Custom optimized Qwen3.6-35B at 4-bit: Around 85 tokens/second generation

The pattern becomes clear: once you're operating at "useful" parameter counts (generally 7B+ for coding tasks), you're hitting a wall regardless of optimization effort.

Why Does This Happen?

The developer who ran these benchmarks suspects memory bandwidth, not raw compute, is the bottleneck. And this makes intuitive sense when you think about how transformer models work.

Every token generation requires reading a substantial portion of model weights from memory. Even with the M4 Max's impressive bandwidth, you're ultimately constrained by how quickly you can move data. Matrix multiplications might be fast, but they're only as fast as the data feeding them.

This explains why smaller models perform dramatically better—there's simply less data to move around. A 0.1B parameter model might hit 1,000 tokens/second, but jump to 1.5B and you drop to around 140 tokens/second. The scaling isn't linear; it's brutal.

What Are Your Options?

If you're looking for coding assistance at 200+ tokens/second while maintaining reasoning and tool-calling capabilities, the landscape narrows considerably:

Cloud Solutions Hosted models from providers like Anthropic (Claude), OpenAI, and DeepSeek offer substantial compute power but come with subscription costs ranging from $20 to $200+ monthly for serious usage. They're reliable and fast, but you're dependent on external services and their availability.

Specialized Hardware Cerebras offers genuinely impressive inference speeds (their GPT-oss-120b model runs at 1000+ tokens/second), but the pricing puts it out of reach for most individual developers and many teams.

Local Models with Adjusted Expectations If you need local inference, consider whether you can accept slightly slower speeds. Models like Qwen3.5-32B or similar architectures at 4-8 bit quantization can provide decent coding assistance at 80-120 tokens/second—enough for productive work if your workflow accommodates slightly more latency.

The Real Question: Local vs. Cloud

This debate ultimately comes down to your specific needs:

• Choose local if you have strict data privacy requirements, intermittent internet access, or want to experiment without ongoing costs
• Choose cloud if consistent speed matters more than ownership, you need the absolute best model capabilities, or your compute budget can support subscriptions

For many developers, a hybrid approach makes the most sense—local models for experimentation and quick tasks, cloud for production workloads where speed and capability are critical.

The Takeaway

Apple Silicon is genuinely impressive for many AI tasks, but running fast local inference with capable coding models remains challenging. The hardware simply wasn't designed with this specific workload in mind, and no amount of Metal optimization can overcome fundamental architectural constraints.

If you're building an AI-assisted development workflow, your best bet is matching your infrastructure to your actual needs—and being honest about whether the "local first" approach serves you or holds you back.

What's your experience with local AI inference? Have you found configurations that break through these performance barriers?