Running AI Coding Agents Locally on Your Mac: A Practical Guide

Why Go Local?

We've all been there. You're deep in a coding session, the creative flow is finally hitting, and then—internet outage. Your cloud-based AI assistant becomes useless, and you're left staring at your code wondering what to do next.

That's exactly what happened to me recently, and it pushed me to finally explore running a capable AI coding agent entirely locally. What I discovered surprised me: with the right setup, you can get surprisingly respectable performance on Apple Silicon, often beating specialized Mac-optimized solutions.

The Setup That Actually Works

After plenty of testing and benchmarking, here's the configuration that delivered the best results on my M1 Max with 64GB of unified memory:

Core Stack:

• llama.cpp built with Metal acceleration
• Gemma 4 26B-A4B in GGUF format (quantized to Q4)
• Multi-Token Prediction (MTP) draft model for speculative decoding
• Gemma 4 multimodal projector for screenshot support
• Pi as the terminal-based coding agent

This isn't the most powerful setup you could theoretically build, but it's the sweet spot for actual usability. You want speed measured in tokens per second, not minutes per response.

The Numbers That Matter

I ran consistent benchmarks using this prompt across all configurations:

"Write a compact Python function that parses a unified diff and returns the changed file paths. Then explain two edge cases."

Each test generated roughly 128 tokens, giving us a fair comparison of generation speed.

Baseline Performance:

Running Gemma 4 directly through llama.cpp with Metal gave us 58.2 tokens per second. That's usable, but honestly? It felt sluggish during actual coding sessions where you're making multiple tool calls and waiting for responses.

The MTP Difference:

This is where things get interesting. Multi-Token Prediction allows the model to "speculate" multiple tokens ahead, accepting correct predictions and rolling back incorrect ones. With the Q8 MTP draft model enabled, performance jumped to 72.2 tokens per second — a 24% improvement without any change to the main model.

The prompt processing stayed essentially the same (around 297-299 tokens/second), but generation speed is what matters for agent workflows. You're not submitting new prompts constantly — you're waiting for the model to generate responses, make decisions, and execute tools.

I tested draft token counts from 1 to 6, and on my M1 Max, 3 draft tokens hit the sweet spot. Values above 4 actually started slowing things down, which makes sense — more speculation means more wasted work when predictions are wrong.

llama.cpp vs. MLX: The Surprising Winner

Here's what caught me off guard: I expected MLX (Apple's native ML framework) to dominate since it's specifically optimized for Apple Silicon. The reality was different.

| Runtime | Generation tok/s | |---------|------------------| | llama.cpp Metal + MTP | 72.2 | | llama.cpp Metal | 58.2 | | MLX-LM (Unsloth 4-bit) | 45.8 | | MLX-LM (standard 4-bit) | 43.9 | | MLX-LM (OptiQ 4-bit) | 38.1 |

Llama.cpp with MTP was roughly 58% faster than the best MLX configuration. The years of optimization work that have gone into llama.cpp have clearly paid off — it runs excellently on macOS despite being cross-platform.

Adding Vision Capabilities

For a proper coding agent experience, you want to be able to send screenshots. Maybe you want the agent to see what it's built, or review a UI change you just made.

The catch? Only Gemma 4 12B is natively multimodal. The 26B model we're using needs the external multimodal projector loaded alongside it.

When I added the --mmproj projector to llama.cpp, it properly advertised multimodal capabilities to Pi, and image tool outputs started flowing through correctly. More importantly, this didn't introduce any measurable slowdown — the generation speed stayed at 72.2 tokens per second.

The Real-World Experience

With this setup, you're looking at roughly 17GB of model files (16GB for the main model, plus the MTP head and projector). For someone with 64GB of unified memory, that's entirely manageable.

Pi as the agent provides a clean terminal interface that hooks into the OpenAI-compatible API that llama.cpp's server mode exposes. This means you can use it with any tool that supports OpenAI's API format, giving you flexibility without lock-in.

The biggest benefit? When your internet fails — and it will, at the worst possible moment — you keep coding. The agent might be slightly slower than cloud alternatives, but it's responsive enough to maintain your workflow.

Getting Started

You'll need to build llama.cpp with Metal support enabled. The project has solid documentation for macOS builds, and once compiled, the server mode gives you that OpenAI-compatible endpoint.

For models, the Unsloth GGUF quantizations on Hugging Face are well-optimized. You'll want the Q4_K_XL quantization for the main model and the matching Q8 MTP draft model.

Tune your --spec-draft-n-max value — start at 3 and test from 1 to 6 to find what works best on your specific hardware. Different Apple Silicon configurations may have different sweet spots.

Is This Worth It?

If you code regularly with AI assistants and have the hardware (16GB minimum, 32GB+ recommended), absolutely. The independence from internet connectivity alone makes it valuable, and with MTP bringing generation speeds into genuinely usable territory, the experience is surprisingly polished.

You're not going to match GPT-4 class intelligence with these open models, but for code completion, refactoring, debugging assistance, and general coding agent tasks? It's more capable than most people expect, and it's yours — running locally, privately, without latency spikes or service outages.

The tools have matured significantly. If you've tried local models before and been disappointed by the speed, give this setup a shot. MTP changes the equation considerably.