Chunking Strategies for Web-Crawled Content in RAG

Most teams building a retrieval-augmented generation system spend their time picking an embedding model, running comparisons, reading leaderboards, arguing about which vendor's vectors are best. That's the wrong fight. A Vectara study peer-reviewed at NAACL 2025 tested 25 chunking configurations against 48 embedding models and found that how you split your documents matters as much as, or more than, which embedding model you choose https://www.premai.io/blog/rag-chunking-strategies-the-2026-benchmark-guide/. Swapping in a better embedding model produces only a marginal lift. Fix a bad chunking setup and the gains run much deeper.

Weaviate's 2025 benchmark puts a hard number on what's at stake. Running the same corpus through the same retriever with the same embedding model produces a recall gap between the best chunking method and the worst as wide as 9% https://denser.ai/blog/rag-chunking-strategies/. That's not a rounding error. A RAG system built on the better method answers questions correctly, while one built on the worse method quietly hallucinates because it retrieved the wrong paragraph.

The failure runs in two directions, and both are common. Chunks that run too large dilute the embedding signal: cram a whole section, with three different subtopics, into one vector, and that vector represents none of them well https://www.premai.io/blog/rag-chunking-strategies-the-2026-benchmark-guide/. Query for one narrow fact and the system pulls back a chunk that's mostly noise, with the fact buried somewhere in the middle. Go too small in the other direction and context is lost: a fragment might match the query on keywords but carry none of the surrounding meaning needed to actually answer it. Web-crawled content makes both failure modes worse, for reasons that have nothing to do with the chunking algorithm itself and everything to do with what a web page actually is.

Web-crawled content before chunking, and its differences from a PDF or a clean prose document

Nearly every chunking guide in circulation was written against tidy input: PDFs, Markdown files, structured internal docs. Feeding a web-crawled page into that same logic breaks it down fast, because a scraped page is nothing like a clean document. It's a pile of structural debris with actual content buried somewhere inside it.

Pulling the raw HTML off a live page brings with it navigation menus repeated in the header and footer, cookie consent banners, ad slots, breadcrumb trails, sidebar widgets, social share buttons. None of that carries meaning. All of it costs tokens. Treat them the same way and one gets over-chunked into meaningless slivers while the other gets under-chunked into an unreadable slab.

Then there's the part a static crawler misses. Modern pages render a meaningful share of their content through JavaScript after the initial page load, and a static fetch misses significant portions of modern pages, without touching what came in dynamically. If the crawl only ever sees the skeleton, no chunking strategy downstream, however smart, can rescue content that was never captured.

The most damaging effect of all this is quieter than it sounds: boilerplate poisoning. If nav links and footer text don't get stripped before chunking starts, they end up embedded exactly like real content, sitting in the vector index next to genuine paragraphs. Query the system later and there's a real chance it hands back a chunk that reads "Home | Products | About" with a straight face, ranked right alongside the passage that actually answers the question. The embedding model didn't fail. The retriever didn't fail. The chunk it was given to work with was garbage before it ever got vectorized.

The pre-chunking step that determines whether any strategy can work: converting crawled HTML to clean, structured Markdown

None of the chunking strategies discussed anywhere in this piece work if the input arriving at the chunker is raw HTML. Converting that HTML into clean, structured Markdown is a required step at the front of the pipeline. It's the gate every downstream strategy passes through or fails at. Skipping it or doing it badly leaves the smartest chunking logic in the world just organizing noise into slightly different shapes.

This conversion step is load-bearing. It's load-bearing. Every strategy discussed later in this piece, recursive splitting, document-aware chunking, semantic chunking, assumes the input has already been through this filter.

Two separate jobs get bundled into "cleaning," and they deserve to be pulled apart. The first is boilerplate removal: stripping navigation, ads, footers, cookie notices, anything structurally present on the page but semantically empty. The second is structure preservation: keeping headings, tables, code blocks, and list hierarchy intact so that a document-aware chunker downstream has something to split on. These two jobs actually pull against each other. Clean too aggressively and the heading hierarchy that document-aware chunking depends on gets flattened along with the ad slots. Clean too lightly and boilerplate survives into the embedding step, poisoning the index the way described above. There's no universal setting here. It's a calibration problem specific to the site being crawled.

Tables deserve their own line of attack, because generic text conversion mangles them reliably. Two tools appear in benchmark results: LlamaParse hits 78% edit similarity at a cost of $0.003 per page https://webscraft.org/blog/chunking-strategies-v-rag-2026-yak-pravilno-rozbivati-dani-dlya-production. Docling, which is open source, hits 97.9% accuracy on table extraction https://webscraft.org/blog/chunking-strategies-v-rag-2026-yak-pravilno-rozbivati-dani-dlya-production. The practical move once a table survives extraction intact: treat it as one whole chunk, no splitting, one chunk equals one table, and run standard text-based chunking only on the prose sections around it. A table sliced mid-row by a generic splitter is worse than useless. It's actively misleading, because a partial row read out of context can look like a complete fact.

Performance of the core chunking strategies on web content specifically, versus the benchmark corpora they were designed for

Recursive character splitting is the default most people reach for first, and for web content, it earns that position. The method works through a priority hierarchy of separators, paragraph breaks first, then sentence breaks, then word boundaries if it has to, stopping at whichever level produces a chunk close to the target size. The FloTorch benchmark, run across 50 academic papers totaling 905,746 tokens, found recursive splitting at 512 tokens scored 69% end-to-end answer accuracy https://www.firecrawl.dev/blog/best-chunking-strategies-rag https://www.premai.io/blog/rag-chunking-strategies-the-2026-benchmark-guide/. First place. Ahead of every more expensive, more clever method tested against it.

On web content specifically, the reason it holds up is almost mundane: paragraph breaks and line breaks are natural signals in Markdown-converted pages, and when those signals are missing or inconsistent, the method degrades gracefully instead of failing hard. It can produce a usable result without the page being well-structured. It just needs some structure.

Where it breaks is heading hierarchy. Recursive splitting has no concept of an H2 boundary versus a stray line break in the middle of a paragraph, it treats them identically. On a page where boilerplate cleaning was incomplete, it will also happily stitch nav remnants right into a content chunk, because it cannot distinguish a real paragraph from leftover footer text." The fix, in practice, is to customize the separator list so Markdown heading markers like \n## and \n### get treated as preferred split points, and to start at 512 tokens with roughly 10 to 20% overlap as the benchmark-backed default https://denser.ai/blog/rag-chunking-strategies/. A January 2026 systematic analysis found overlap added no measurable benefit when paired with SPLADE retrieval, so pipelines running sparse retrieval can skip it and save on index size.

Document-aware chunking is the natural fit once a page has real structure to work with. Instead of splitting by character count, it parses the document's actual elements first, headings, sections, tables, code blocks, and uses those as chunk boundaries rather than cutting wherever the token count runs out. On a Markdown-converted page with intact heading hierarchy, this is exactly the shape of input the method was built for: an H2 boundary is a genuinely coherent split point, not just a line break that happens to occur near one.

Document-aware chunking depends entirely on what arrives before it. If the Markdown conversion collapsed the heading hierarchy into flat, undifferentiated text, there's nothing left for this method to key off of, and it loses its entire advantage over recursive splitting. It shines on documentation sites, product pages with clear section breaks, and editorial content with a real H2/H3 skeleton. It does much less for shallow pages, product cards, search result listings, anything without a structural backbone to parse in the first place.

Semantic chunking is the most seductive option on paper and the riskiest in practice. It groups sentences by embedding similarity and cuts a new chunk wherever that similarity drops below a set threshold. Chroma's benchmark clocked it at 91.9% retrieval recall, a genuinely strong number https://www.premai.io/blog/rag-chunking-strategies-the-2026-benchmark-guide/. Compared against FloTorch's 69% for recursive splitting, it looks like semantic chunking should win outright. It doesn't, because the two numbers measure different things: one is recall, whether the right chunk got retrieved at all, and the other is end-to-end answer accuracy, whether the system actually answered the question correctly using what it retrieved.

The threshold setting is where semantic chunking either works or quietly falls apart. Set it below 0.75 and unrelated topics start bleeding into the same chunk. Setting it above 0.85 causes the text to get excessively fragmented https://webscraft.org/blog/chunking-strategies-v-rag-2026-yak-pravilno-rozbivati-dani-dlya-production. Web pages are an especially hostile environment for this kind of miscalibration, because it's routine for a single page to cover a pricing table, a technical spec sheet, and a paragraph of marketing copy all in the same scroll. A wrongly tuned threshold on that kind of page either merges pricing and specs into a muddled chunk or fragments the spec sheet into pieces too small to answer anything.

The most concrete failure on record: FloTorch's 2026 run found semantic chunking producing fragments averaging just 43 tokens https://www.premai.io/blog/rag-chunking-strategies-the-2026-benchmark-guide/. A 43-token chunk isn't a chunk, it's a scrap (the research suggests avoiding fragments much below 100–150 tokens), since semantic chunking without a floor is not safe for web content where shallow sections are common. Semantic chunking without that floor is not safe to run against web content, where thin, shallow sections appear constantly. The Vectara NAACL 2025 study backs up the caution with a broader finding: fixed-size chunking consistently beat semantic chunking across document retrieval, evidence retrieval, and answer generation. The extra computational cost of running embeddings just to find split points often isn't earning its keep on realistic document sets. This maps naturally onto page structure (a subsection (H3 + its paragraphs) becomes the child chunk, while the full H2 section becomes the parent chunk).

The freshness problem that web-crawled RAG introduces and generic chunking strategies ignore

A PDF doesn't change. Neither does an internal manual, once it's uploaded. Web content does, constantly, and that single fact breaks an assumption baked into most chunking advice. Every strategy discussed above, recursive, document-aware, semantic, was designed against a static corpus that gets chunked once and stays chunked. Live web data doesn't hold still long enough for that to work.

Pricing changes. Product specs update. An article gets a correction. That kind of change appears not as a new document, but as a quiet edit to a page that's already been crawled, chunked, and embedded. If the pipeline has no way to detect that edit and re-chunk just the affected section, the index keeps serving a confident answer built on data that's already wrong. Choosing a chunking strategy for web-crawled RAG means choosing it alongside an update architecture, deciding how often pages get re-crawled, how changes get diffed against what's already indexed, and how stale chunks get retired rather than left to rot next to fresh ones. Get the chunking boundaries right and skip this part, and the system will still confidently retrieve last month's price. In the traditional model, 20% of time was spent building scrapers and 80% maintaining them https://tendem.ai/blog/future-of-web-scraping-ai-agents-human-co-pilots. Usage of AI-driven discovery for job listings content surged over 50x at Zyte in 2025 https://www.zyte.com/blog/ai-is-the-new-engine-for-web-scraping/. Page-level chunking won NVIDIA's 2024 benchmarks with 0.648 accuracy and the lowest variance https://www.firecrawl.dev/blog/best-chunking-strategies-rag. Semantic chunking can improve recall by up to 9% over simpler methods https://www.firecrawl.dev/blog/best-chunking-strategies-rag.