terms.txt: The Missing Contract Between AI Agents and the Open Web

Let's be honest: the web has a crawling problem, and it's getting worse.

For decades, websites operated on a gentleman's agreement with search engines. We'd let their bots crawl our pages, and they'd send visitors back. It wasn't formal, but it worked. Your friendly robots.txt file was essentially a "please be nice" note stapled to the door.

That era is ending fast.

The Bots Are Winning (And Not in a Good Way)

Automated traffic now dominates the internet. According to researchers analyzing web traffic patterns, AI crawlers and agents make up the majority of requests hitting major CDNs. Here's the uncomfortable reality:

• AI platforms fetch thousands of pages for every single visitor they return to a website
• Training data collection has become the dominant form of web crawling
• Traditional robots.txt was never designed for this world—it can't verify identity, specify purpose, define terms, or set prices

It's like showing up to a restaurant and finding 10,000 people eating for free because they vaguely promised to maybe send a few customers your way someday.

Why robots.txt Can't Cut It Anymore

Let's appreciate what robots.txt actually is: a voluntary, honor-system text file that crawlers can ignore. It's about as legally binding as a "quiet zone" sign at a loud rock concert.

The newer alternatives? Mostly proprietary CDN features that lock you into specific providers. If you're not using Cloudflare, Akamai, or whatever enterprise solution du jour, you're out of luck.

For the independent developer or startup running their own infrastructure, you essentially have no standardized way to negotiate with AI systems accessing your content.

Enter terms.txt: A Smarter handshake

A new proposal from researchers at arXiv (arXiv:2609.11152) introduces terms.txt—think of it as robots.txt grown up and ready for the real world.

The core idea: a machine-readable terms file that websites can place alongside their content, expressing:

• Who is accessing (identity verification via Web Bot Auth signatures)
• Why they're accessing (signed intent declarations)
• What they can access (per-path, per-purpose rules)
• How much (optional: HTTP 402 negotiation for actual payment)

The system includes delegation tokens so AI agents can act on behalf of developers, signed receipts so websites can audit actual access, and origin-enforced cryptographic guarantees.

The Technical Highlights

The beauty here is in the details:

• Dependency-free implementation: Adds only 0.20 to 0.65ms overhead on a single vCPU. That's essentially invisible for most use cases.
• No vendor lock-in: Unlike CDN-specific solutions, this works at the origin server level.
• Graduated complexity: You can start simple (like robots.txt) and add sophisticated terms as needed.

Why Developers Should Care

If you're building AI-powered tools, scraping data, or running any service that pulls content from the web, this matters for several reasons:

1. Compliance gets standardized

Instead of negotiating access agreements one-by-one, imagine a world where sites simply publish their terms and you sign them cryptographically. Much cleaner.

2. New revenue models emerge

HTTP 402 ("Payment Required") has existed for years but never had infrastructure to support it. Now websites can actually charge for premium API-style access to their content—per-request microtransactions, anyone?

3. Respect becomes verifiable

For ethical AI builders, this provides proof you're honoring site preferences. A cryptographically signed intent to comply beats "we promise we read robots.txt."

The Road Ahead

Let's not get ahead of ourselves. This is a research proposal, not a standard. Adoption would require buy-in from major AI providers, browser makers, and the broader development community.

But the timing is right. The web's informal scraping economy is breaking down, legal battles over AI training data are multiplying, and there's genuine appetite for standards that don't require enterprise CDN contracts.

At NameOcean, we see infrastructure debates like this one shaping how the web evolves. Whether you're registering domains, hosting applications, or building the next generation of AI tools, understanding these protocol-level conversations helps you anticipate where the internet is heading.

The bots aren't going away. The question is whether we build a fair system for managing their access—or keep pretending a text file that everyone can ignore is sufficient.

What do you think? Is formal negotiation between websites and AI systems inevitable? Drop your thoughts below.