The Web's Scraping Problem Is Spiraling Out of Control

Let's cut to the chase: the internet has a serious issue with how bots consume content, and it's getting uglier by the day.

For years, websites and search engines operated on a kind of unwritten deal. We let their crawlers扫荡我们的页面, they sent traffic our way. Nobody signed anything. Nobody negotiated terms. It just... worked. Your humble robots.txt file was basically a sticky note on the door saying "please be cool."

Those days are numbered.

The Bots Are Taking Over (And Not in a Good Way)

Automated traffic is now eating the internet alive. Researchers analyzing network patterns at major CDNs have found something uncomfortable: AI crawlers and agents now make up the bulk of requests hitting servers.

Here's what that looks like in practice:

• AI platforms will rip through thousands of pages just to send one actual visitor your way
• Training data harvesting has become the primary driver of web crawling activity
• The old robots.txt standard wasn't built for this—it can't verify who's knocking, why they're knocking, or what they plan to do with your content

Imagine running a restaurant where 10,000 people show up to eat for free because they vaguely promised to maybe recommend you to a few friends.

Why robots.txt Doesn't Cut It Anymore

Let's give robots.txt its due: it's a voluntary, honor-system text file that crawlers can completely ignore. It's about as enforceable as a "quiet zone" sign at a metal concert.

The "solutions" that have emerged since? Mostly proprietary CDN features that chain you to specific providers. If you're not paying for Cloudflare, Akamai, or some enterprise platform, you're left out in the cold.

For indie developers or startups running their own setups, there's essentially zero standardized way to negotiate with AI systems hitting your pages.

Introducing terms.txt: A Real Handshake

Researchers have floated a new proposal (arXiv:2609.11152) that basically asks: what if robots.txt grew up and faced reality?

The concept is a machine-readable terms file that sites can serve alongside their content, laying out:

• Who is accessing (identity verification through Web Bot Auth signatures)
• Why they're accessing (signed intent declarations)
• What they can access (per-path, per-purpose rules)
• How much (optional: HTTP 402 payment negotiation for actual compensation)

The system supports delegation tokens so AI agents can operate on a developer's behalf, signed receipts for auditing real access patterns, and cryptographically enforced guarantees that actually mean something.

What's Actually Impressive

The implementation details matter:

• Barely any overhead: Adds just 0.20 to 0.65ms on a single vCPU. Basically imperceptible.
• No vendor prison: Works at the origin server level—no CDN lock-in required.
• Start simple, get sophisticated: You can begin with basic rules and layer on complex terms as your needs grow.

Why This Should Matter to You

If you're building AI tools, scraping data, or running any service that pulls content from the web, here's why you should pay attention:

1. Compliance becomes standardized

Forget negotiating individual access agreements. Picture a world where sites publish their terms and you sign them cryptographically. Way cleaner.

2. Actual monetization becomes possible

HTTP 402 ("Payment Required") has existed forever but never had the infrastructure to back it. Now sites can genuinely charge for premium API-style access to their content. Per-request payments, anyone?

3. Good faith becomes provable

For ethical AI builders, this gives you cryptographic proof that you're respecting site preferences. A signed declaration of compliance beats "trust us, we read robots.txt."

What's Next

Let's not get carried away. This is a research proposal, not an established standard. Widespread adoption would need buy-in from major AI providers, browser developers, and the broader ecosystem.

But the timing feels right. The informal scraping economy is collapsing under its own weight. Legal battles over AI training data are multiplying. And there's real demand for standards that don't require enterprise CDN contracts.

At NameOcean, we watch these infrastructure-level debates closely because they shape how the web evolves. Whether you're registering domains, hosting applications, or building the next wave of AI-powered tools, understanding these protocol conversations helps you see where the internet is heading.

The bots aren't disappearing. The real question is whether we build a fair system for managing their access—or keep pretending a text file that anyone can ignore is good enough.

What are your thoughts? Is formal negotiation between websites and AI systems inevitable? Share below.