How Coding Agents Are Rewriting the Rules of AI Infrastructure (And What It Means for Your Next Deployment)

• Introduction to the problem
• What TraceLab is
• Key findings and observations
• Implications for infrastructure/serving
• Future directions
• Conclusion

The AI coding assistant arms race has officially entered a new phase. With Claude Code, Codex, and Gemini CLI now battle-tested by developers worldwide, the question is no longer whether these tools work—it's how to serve them at scale without breaking the bank or introducing latency that makes developers want to throw their keyboards out the window.

A fascinating new research paper from the University of Washington's SyFI lab drops some hard numbers on what real coding agent workloads actually look like, and the results might surprise you.spoiler: your existing LLM serving infrastructure probably isn't optimized for this.

The Autonomous Revolution Nobody Prepared For

Here's the thing about coding agents that seems obvious once someone says it out loud: they don't just answer questions. They execute plans. When you ask an AI coding assistant to implement a feature or fix a bug, you're not initiating a simple query-response transaction—you're kicking off an autonomous workflow where the model will reason, execute shell commands, read and modify files, run tests, and iterate until it's satisfied (or until you interrupt it).

The research team captured over 4,300 real-world sessions totaling roughly 55 billion tokens, and the patterns are striking. Each user request triggers an average of 8.8 LLM-tools cycles before completion. Think about that: for every question a developer asks, there are nearly nine rounds of model generation followed by tool execution. The model isn't just talking—it's doing. And 88% of all LLM inference rounds happen not in response to user input, but in response to tool results.

This has massive implications for throughput planning. Traditional chatbots are essentially request-response systems with occasional follow-ups. Coding agents are more like long-running autonomous processes with frequent context-switching between generation and execution phases. Your autoscaling policies, your request queuing, your caching strategies—all of these need to be rethought through the lens of autonomous multi-step workflows rather than simple turn-taking conversations.

The Input-Output Asymmetry That's Breaking Your Cache

Here's a number that should make every infrastructure engineer do a double-take: inputs outnumber outputs by 294 to 1. In the traces analyzed, models read 52.56 billion cached input tokens and prefilled 2.34 billion new ones, yet generated only 186.9 million output tokens. A typical round sits on a 32K to 256K token prefix and adds just a few hundred to a few thousand tokens before decoding a couple hundred out.

This creates a fundamentally different performance profile than what most LLM serving systems are optimized for. Traditional language model inference focuses heavily on generation throughput—how many tokens per second can we produce? But for coding agents, the bottleneck often isn't decoding speed; it's time-to-first-token (TTFT) after tool execution returns. The team found that Codex's TTFT for each step sits around 3.1 seconds, which represents roughly 25% of a round's total generation time. When you're doing nine rounds per request, that overhead compounds fast.

The caching story is also more nuanced than it first appears. Yes, input caching (prefill caching) helps with the massive context that coding agents carry. But the prefill workload is split between cached tokens and new tokens that need actual computation. Short, incremental prefills after tool execution have different performance characteristics than the initial long-context load. Serving systems need to handle both efficiently, and the optimal strategies may differ between these cases.

Shell Commands Are Eating the World (Still)

If you thought coding agents would be doing sophisticated reasoning and complex refactoring, you weren't wrong—but the reality is messier. Of the 433,000 tool calls captured in the traces, 76% were shell or command executions. Running builds, tests, git operations, file manipulations through CLI tools—the unglamorous stuff that makes up the actual daily workflow of software development.

File edits came in at 11%, file reads and searches at 9%, with planning, sub-agent delegation, and web lookups making up the remainder. This concentration in shell commands matters for several reasons. First, these are long-tailed operations: a simple ls is fast, but running a full test suite or compiling a large codebase can take minutes. The latency distribution spans four orders of magnitude, which makes capacity planning a nightmare.

Second, the tool vocabulary varies significantly between agents. Claude Code drew on 54 different tools while Codex used 31, but both concentrated the vast majority of their volume in the same core set—shell, file edits, file reads. This suggests that optimizing for these high-frequency tools could yield outsized improvements, even without tackling the full tool ecosystem.

What This Means for Your Infrastructure

If you're building or operating AI-powered development tools, this research points to several concrete takeaways.

Think in workflows, not requests. Your serving infrastructure needs to account for the autonomous, multi-step nature of coding agents. This means longer-running request contexts, more sophisticated state management, and batch scheduling opportunities that group related tool calls together.

Optimize for the idle gaps. Users spend an average of 46.7 minutes between requests, despite a median think time of only 1.4 minutes. The distribution is extremely heavy-tailed—some developers iterate rapidly while others start a request and come back hours later. Smart prefetching and re-prefill strategies during these idle periods could dramatically reduce perceived latency without increasing compute costs.

Separate your prefill and decode paths. The 294:1 input-output ratio means your prefill computation is likely your dominant cost. Incremental prefills after tool execution need different optimization than initial context loading. Consider separate resource pools or scheduling policies for these different workload types.

Don't forget the tool execution infrastructure. When 76% of agent activity is shell commands, your container orchestration, filesystem access patterns, and build tool caching become first-class concerns for AI agent serving. The model inference is just part of the pipeline.

The Road Ahead

The TraceLab team has open-sourced their trace collection and analysis pipeline, making it possible for infrastructure teams to generate their own traces and optimize for actual usage patterns rather than synthetic benchmarks. This kind of real-world data is exactly what the AI infrastructure space needs as it matures from "make it work" to "make it work at scale."

For the NameOcean Vibe Hosting community, these insights highlight why we're investing in infrastructure that goes beyond simple GPU allocation. Serving AI coding agents isn't just about raw compute—it's about understanding the unique characteristics of agentic workloads and building systems that can handle the autonomous, multi-step, tool-heavy reality of how developers actually use these tools.

The era of the coding agent is just beginning, and the infrastructure decisions we make today will shape how efficiently these tools serve developers for years to come.