Converting Raw HTML to Clean Markdown for LLM Ingestion

Open a scraped page in a text editor and scroll past the first screen of output. Most of what fills that screen looks like this: <div class="flex items-center justify-between px-4 py-2 bg-gradient-to-r from-blue-500 to-purple-600">. It carries no information about the subject of the page. It's a styling instruction meant for a browser's rendering engine, and it is sitting in the same file, read by the same model, as the sentence it wraps around.

Raw HTML is a weak default input for a language model because most of its tokens carry no meaning the model can use. A typical webpage runs 50,000 to 200,000 tokens of raw HTML, while the actual content on that page, the article text, the product details, the documentation, is usually a small slice of that total. The rest is opening and closing tags for every element, class names and inline styles, whitespace and comments, embedded JavaScript and CSS, meta tags, noscript blocks, and SVG icons, all billed at the same per-token rate as the words that matter. A language model has no way to skip this filler. It reads every tag, reasons over every attribute, and gets billed for every token, whether that token is a sentence from the article or a leftover class name from a design system.

The cost appears twice: once at the token meter, and again inside the model's understanding of the page. When an embedding model processes a chunk of text dominated by CSS classes and HTML attributes, the vector it produces ends up representing markup structure more than it represents the actual information on the page, and retrieval in a RAG system suffers as a result. This isn't a bug that a smarter parser will eventually fix. Modern web pages are built to be rendered by a browser, not read by a model, and the gap between what a page looks like and what its HTML actually contains keeps getting wider as frameworks add more structure around less content.

Why stripping tags is not enough

The natural fix is to strip the tags and keep whatever text is left. That instinct points the right direction, but the execution leaves behind a different kind of mess, trading one category of noise for another. Most people reach for innerText, a regex pass, or BeautifulSoup's .get_text(), and for the first few pages it looks like it solved the problem.

Then the output starts to misbehave. Headings stop looking like headings and blend into the paragraph beneath them, so a model reading the result has no way to tell a section title from supporting detail. Lists turn into flat runs of sentences, losing the signal that a set of items belongs together as parallel options rather than a continuous thought. Links vanish outright, even on pages where the link text and its destination were the entire point of that line. Tables turn into rows of words with no sense of which value belongs to which column.

At the same time, tag-stripping keeps plenty of material that never should have survived. Navigation bars, footers, and cookie banners are visual affordances with no informational content once they're rendered as plain text. Icon labels like navigate_before, chevron_left, and expand_more appear in the extracted text as stray words, even though in a browser they were meant only to be displayed as symbols, not read. Framework internals leak through too: Tailwind class names appearing as text content, Next.js hydration code, stray @keyframes and @font-face declarations, none of it part of the page's actual message. Pages also repeat themselves in ways a human skimming the browser never notices: the same call-to-action appearing four times, carousel content duplicated for mobile and desktop breakpoints, sticky headers getting re-extracted partway through the page.

Stripped text, in the end, is a wall of words with no hierarchy. The model gets content, but content without structure isn't clean content; it's just less of the wrong kind of noise. The real task has two separate parts: removing material that never should have been extracted, and preserving the structural signal, headings, lists, columns, that tells a model how the remaining words relate to each other. Tag-stripping fails at both, because it was never built to handle either on purpose.

Why Markdown is the right intermediate format

Markdown solves both halves of that problem at once. It keeps the semantic structure that tag-stripping throws away while cutting out the syntax that makes raw HTML so expensive to feed a model. A heading becomes a line starting with #. A list becomes a line starting with - or *. A link becomes text(url). Paragraphs separate with a blank line. Every hierarchical relationship the HTML encoded is still there, just written in a fraction of the tokens.

The savings compound. A typical HTML article can run several times the token count of the same content written in Markdown, and that multiplier applies at every single call through a pipeline, not just once at ingestion. Part of the reason is that large language models read Markdown the way they were trained to, so the overhead of a ## or a - is small next to something like <div class="container mx-auto">.

The benefit reaches past cost and into accuracy. Clean Markdown gives an embedding model text where the signal is high and the noise is low, so the resulting vectors represent what the page actually says rather than how it was styled, and retrieval quality in a RAG system improves directly because of it. Markdown also keeps the one structural feature that matters most for retrieval: heading hierarchy. A retriever that can see which lines are section titles and which are supporting detail can match a query to the right part of a document, a distinction flat text simply cannot offer.

The rendering problem that library-based conversion cannot solve alone

Turning a static HTML page into Markdown is a solved problem. Turning the modern web into Markdown is not, because most pages today don't ship their content in the HTML at all, they build it in the browser after the page loads, and a plain HTTP fetch only catches the page mid-load, before any of that content exists.

Most single-page applications serve an empty <div id="root"> and fill it in with JavaScript after the page arrives in the browser. Fetch one of these pages with a standard HTTP client and a Markdown converter will return close to nothing, even though the full rendered version of that same page might represent 50,000 to 200,000 tokens of HTML once the browser finishes building it. The only way to see what a user actually sees is to run a headless browser: load the page, execute its JavaScript, wait for its network requests to settle, then pull the final computed DOM. Only at that point is there enough HTML to convert into something accurate.

Running headless browsers at scale carries real costs of its own. A team has to manage a fleet of headless Chrome instances, catch memory leaks before they crash a process, and cap how many pages run at once so the whole system doesn't fall over. Beyond the browser itself, there are gatekeepers on the other end: rate limits, automated traffic detection, CAPTCHA challenges. A pipeline that can't get past these simply runs out of pages to convert. A conversion library handles the easy pages fine and stumbles on the pages that matter most, news sites, SaaS documentation, product catalogs, because a large share of those run on React or Next.js and render their content client-side.

What a production-grade conversion pipeline does, stage by stage

A real HTML-to-Markdown pipeline isn't one conversion step, it's a sequence of separate operations, each one built to fix a specific kind of problem, and skipping any single stage lets back in the noise the other stages were built to remove.

The first stage is rendering. A headless browser, Playwright or something comparable, loads the page, runs its JavaScript, and waits for network activity to settle before any text gets pulled out. This step exists to make sure the DOM being converted matches what a person would actually see on their screen, not some half-loaded placeholder. Pages that load slowly or run heavy JavaScript sometimes need the wait time pushed up to 3,000 or even 5,000 milliseconds to capture everything, though that extra patience costs throughput and money.

The second stage is sanitization, where the rendered DOM gets pruned before any conversion happens. A solid extraction step scores each node in the DOM by its text density, its ratio of links to actual text, and whether it sits inside a semantic tag like <main> or <article>, then strips out navigation bars, footers, sidebars, modals, and ad slots based on those scores. When this stage is skipped, the Markdown that comes out the other end carries navigation links, newsletter signup forms, related-article previews, and cookie banners sitting at the same weight as the article itself.

The third stage is the conversion itself, where the now-sanitized DOM gets translated line by line into Markdown, headings, lists, code blocks, tables, and links each mapped to their Markdown equivalent.

The fourth stage is where most of the remaining cost gets cut. A standard conversion library only translates HTML structure into Markdown syntax, it doesn't know what a model actually needs, so a second pass has to handle several more jobs. Image handling collapses clusters of logos into a single line, turns linked images into plain links, strips out decorative images that carry no information, and keeps alt text where it actually describes something, since on a typical marketing page images can make up a large share of the output with zero informational value. Emphasis cleanup removes bold and italic markers used only for visual weight rather than actual meaning, since **Get started today** and plain Get started today tell a model the exact same thing. Link deduplication pulls every link out of the inline text and collects it into one list at the end, filtering out navigation links, anchor links, JavaScript void links, and action links, a step that alone can cut 20 to 30 percent of the token count on a link-heavy page. Deduplication beyond links merges headings that just repeat the paragraph below them, collapses carousel content duplicated across mobile and desktop breakpoints into one copy, and reduces repeated phrases down to a single instance. A stat-merging pass reconnects numbers from their labels when a marketing page splits them across separate lines, cleaning up the leftover whitespace in between. A final cleanup pass removes icon labels, cookie consent text, "your browser does not support video" messages, and breadcrumb separators, all of it with nothing to say to a reader or a model.

What comes out the far end is what open-source crawlers built for AI pipelines expose as fit_markdown, boilerplate stripped, main content only, as distinct from raw_markdown, which is the full page after conversion but before optimization.

What the token savings mean for RAG pipeline economics

Running all four stages instead of stopping at basic conversion isn't a small efficiency win, it changes what a RAG system can afford to do and how it gets built, which is the kind of problem Context, a web scraping and crawling API that converts any URL to LLM-ready Markdown for AI pipelines, was built to handle. Microlink has put a figure on this directly, stating that Markdown use cuts token usage by 80 percent compared to raw HTML.

That reduction touches every layer of a RAG pipeline at once, including ingestion, retrieval, and generation. Cutting the token count of the source material by a wide margin saves tokens not once but at every one of those steps, for every document, on every query a system ever runs.

That changes what's affordable to build. A RAG system can therefore pull in more supporting material per query without paying more for it. Embedding costs drop in proportion to the token count, and since embeddings get computed once per document but queried repeatedly over a document's lifetime, the saving at ingestion keeps paying off for as long as that document stays in the index. Latency improves too, since a model processing fewer tokens per chunk returns an answer faster, which matters directly to anyone running a production system at real user volume.

None of this arrives from a single clever trick. It's the output of running rendering, sanitization, conversion, and post-processing as four distinct, deliberate stages, each one removing a category of waste the others were never built to catch. If any one of those stages is treated as optional, the savings shrink back toward whatever a basic library alone can deliver, which, as the earlier sections showed, still leaves plenty of noise sitting in the token count a model has to read, reason over, and get billed for.