Why Your AI Is Only as Good as What It Actually Reads

Here's something most people in the AI space don't like to admit: every time you query a language model, there's a good chance it's reading a lot of garbage.

This happens twice in a model's lifecycle. The first time is during training—that massive scrape of the internet where the model builds its understanding of how language works. The second is during inference, when you ask it a question and it pulls in external context to formulate an answer. In both cases, a significant portion of what gets processed has nothing to do with what you're actually asking.

Recent research puts some concrete numbers on this. On a typical webpage, roughly 70% of the HTML is pure boilerplate—navigation bars, ad units, sidebar widgets, cookie consent popups, footer links. The stuff you actually care about? That's the remaining 30%. And here's what matters: that 30% is where everything happens.

The Real Competitive Advantage Nobody Talks About

The machine learning world loves to obsess over flashy things. Model architectures. Parameter counts. Training methodologies. But a growing body of evidence suggests the unglamorous work of cleaning your data might be where the real advantages hide.

Take a study from 2025 by Ma and colleagues. They took identical Common Crawl snapshots and built two separate training datasets. The only difference? How they extracted the content. One dataset used traditional rule-based methods; the other used a model-based approach.

Everything else stayed the same—same base model, same training duration, same hyperparameters.

The model trained on cleaner, model-extracted data scored 1.08 percentage points higher across 13 different benchmarks. That might not sound dramatic, but consider this: that single change let them outperform models trained on FineWeb and RefinedWeb—two datasets famous for their elaborate filtering pipelines.

Let that sink in for a moment. While everyone races to build bigger models or hunt for novel architectures, the real breakthrough might just be removing the junk.

What Happens When Your Retrieval Gets Messy

Training quality isn't the only concern. Research from 2023 showed something troubling: a single irrelevant passage in your context can completely throw off a model's response. When your retrieval system grabs sidebar ads and cookie notices alongside your actual content, you're not just wasting computational resources—you're actively making your answers worse.

This is a particular headache for production RAG systems. Retrieval-Augmented Generation sounds great in theory—pull relevant information, ground your responses in facts. But if that retrieval includes navigation menus and promotional banners, you're essentially poisoning your context window.

The numbers get really interesting when you look at how different extraction methods handle structured content. Traditional rule-based tools like Trafilatura score around 0.13 similarity for code blocks and 0.61 for mathematical formulas compared to 0.91 and 0.94 for model-based approaches. If your documentation includes code examples or technical notation—and whose doesn't?—these extraction failures corrupt your training data in ways that are hard to detect but easy to feel.

Two Schools of Thought on Extraction

When it comes to actually pulling useful content from web pages, the field splits into two main approaches.

Rule-based extractors work by analyzing surface features—where elements sit in the DOM tree, what HTML tags they use, how dense the text is. Tools like Trafilatura, Readability, and Boilerpipe fall into this camp. They're fast, they don't require machine learning infrastructure, and they're reasonably good at what they do. But they have a fundamental limitation: they judge books by their covers. A navigation table and a data table look identical to these systems structurally, so misclassification is inevitable.

Reading-based extractors take a different path entirely. Instead of analyzing HTML structure, they feed the actual content through transformer models and let the model decide what matters. The current gold standard, Dripper, uses a decoder architecture that generates extraction labels token by token. The accuracy is impressive, but there's a catch: each token requires loading the entire model into memory. Speed becomes a function of memory bandwidth, not raw computing power.

This is where a newer approach called Pulpie enters the picture with an interesting architectural bet. Rather than using a decoder that generates labels sequentially, Pulpie employs an encoder that classifies every HTML block in a single pass. This shifts the bottleneck from memory bandwidth to pure compute—a much more tractable problem at scale.

The performance gap is striking. On an NVIDIA L4 GPU, the smaller Pulpie model processes 13.7 pages every second. Dripper? 0.68. That's roughly a 20x improvement in throughput. Translate that to actual dollars: cleaning a billion pages costs about $7,900 with Pulpie versus $159,000 with Dripper. That's not a marginal improvement—that's an order of magnitude.

What This Means for Your Projects

If you're building AI-powered applications, this research has some practical takeaways worth considering.

Start by examining your extraction pipeline. If you're still using rule-based extractors for training data or production RAG systems, you're accepting measurable performance losses. The cost of better extraction has dropped dramatically over the past year or so. There's really no excuse anymore.

Think carefully about where extraction sits in your architecture. A lot of teams treat web content as a solved problem—fetch the page, strip the HTML, move on. But as models grow more capable, the quality of their inputs becomes increasingly important. Your retrieval system is only as strong as the layer that extracts content from raw pages.

Explore open-source options. Pulpie joins an expanding ecosystem of freely available extraction models. The era of needing expensive proprietary APIs for decent extraction quality is fading fast.

The Underlying Shift

What's really happening here is a change in how the ML community thinks about data. For years, the conventional wisdom held that more data beat cleaner data. Scale your corpus, apply basic filters, train longer—eventually the signal would drown out the noise.

That assumption breaks down once you push into higher capability tiers. When you're training on hundreds of billions of tokens, even small corruption rates accumulate into substantial damage. And when your inference context windows are fixed, every noisy token is one less useful token taking up space.

The irony is that web extraction—the unglamorous work of figuring out which parts of a page actually matter—might be one of the highest-leverage interventions available in your entire ML pipeline. The next meaningful improvement in your model's performance might not come from a revolutionary new architecture. It might come from simply reading the web more carefully.

Clean data isn't exciting. But it's the difference between a model that works and one that truly works.