Converting Raw HTML to Clean Markdown for LLM Ingestion

The first instinct almost every developer has is to grab BeautifulSoup, call get_text(), and call it done. Or use innerText. Or write a regex that yanks out anything between angle brackets. All three get rid of the clutter, sure. But they take the structure down with it, and structure is most of what makes text usable to a model in the first place. This is the wrong call, full stop, and it's worth being blunt about why.

Here's what a tag-strip actually throws away:

• Heading hierarchy. An h1 and an h6 both turn into the same flat paragraph, so the model has no way to tell a title from a footnote.
• Lists collapse into run-on prose, losing every cue that told the reader, or the model, this was a sequence and not a sentence.
• Code blocks lose their formatting entirely, which is close to disastrous if the page is technical documentation.
• Links vanish outright, and on a lot of pages the links were the actual information the page existed to convey.

The output is shorter, yes. But it's shorter in a way that costs meaning, not noise. A model reading tag-stripped text can't tell a section heading from a stray sentence, or a fenced code sample from a paragraph describing that code. It's all just characters at that point.

One fair pushback: none of this matters much at training time. Models never see raw HTML during pretraining, that corpus gets cleaned long before it hits a training run. The pain shows up at inference time, specifically inside retrieval-augmented generation and agent pipelines, where a model has to reason over content pulled in live, on the spot, query by query. Everything from here on is about that inference-time problem, because that's where bad conversion gets felt on every single request.

What a well-designed conversion pipeline actually does

A conversion pipeline that works runs in three stages, and skipping any one of them shows up downstream.

Clean comes first. Before any conversion happens, the pipeline has to isolate the actual content zone of the page and cut everything else: navigation, sidebars, footers, cookie banners, inline scripts, stylesheets. This step alone decides whether the output is usable.

Convert comes next. The cleaned HTML turns into Markdown that respects structure: headings map to a # hierarchy, lists become real enumerations, code gets fenced properly, links stay intact.

Structure is the last mile. Metadata gets attached, heading levels get normalized so a page that starts at h3 doesn't confuse a downstream chunker, and the output gets prepped for whatever indexing or retrieval system sits on top of it.

A reference implementation that's been widely discussed since its release in early 2024 makes this concrete. It runs headless Chrome to fetch a page's actual rendered source, hands that structure to Mozilla's Readability library to strip out headers, footers, navigation, and sidebars, then runs the cleaned HTML through Turndown, which converts it to Markdown using a rule-based, regex-driven approach. Out comes a Markdown file ready to be grounded, summarized, or reasoned over directly.

That pattern handles the page most people picture when they think about scraping: static documentation, blog posts, news articles. It falls apart in two specific spots, and both deserve naming rather than a wave of the hand. Certain interactive web apps and script-heavy sites hand back empty <div> elements to a plain HTTP GET; a real browser has to run the underlying scripts before there's any content to pull at all. And anything sitting behind a login, a paywall, or a session token needs browser state that persists between requests, something a stateless scraper can't hold onto.

A few habits separate a good pipeline from a fragile one. Strip <script>, <style>, <nav>, <footer>, <header>, and <aside> before conversion starts, not after: cleaner input makes cleaner Markdown every time. Normalize whitespace on the way out, collapsing repeated blank lines and trimming trailing spaces so the output doesn't carry visual noise into the token count. And check the output before it enters a retrieval index. Some teams run several passes over the extracted Markdown, each one aimed at a single category of leftover noise, a level of care that only starts to pay off once volume gets high enough to justify it.

Six Python libraries for HTML-to-Markdown conversion and when to use each

No single library wins every case, but they are not interchangeable either. Six of them cover the realistic option space, and picking between them comes down to what the job needs, not which one shows up most in blog posts.

trafilatura is built for web scraping and LLM preprocessing specifically, and it's very good at pulling clean content while throwing out boilerplate. It's fast, pulls metadata alongside content, ships a CLI, and sees active development. Needs Python 3.6 or newer plus lxml. Reach for it when building LLM training datasets, aggregating content, or pulling news articles and research papers at volume. For most teams building a RAG pipeline from scratch, this should be the default, not markdownify.

markdownify trades speed for control. It's subclassable, so custom tag handlers are easy to write, and it drops neatly into an existing BeautifulSoup pipeline. Needs Python 3.7+ and BS4, and it runs slower than trafilatura. Pick it when a project needs domain-specific rules or unusual formatting that a general-purpose tool won't handle out of the box, not as a default first choice.

html-to-markdown targets production systems that care about type safety. Full HTML5 support, real type hints, solid table handling, a CLI, and active maintenance. Needs Python 3.9+ and lxml.

html2text is the no-frills option: no dependencies, tested for years, plenty of configuration knobs, broad platform support. It runs at medium speed and does no content extraction or metadata extraction, and there's no CLI or active development behind it anymore. Pick it when "just works reliably" matters more than features, and accept that it's a dead end for anything needing metadata.

domscribe offers full HTML5 support with decent table handling and custom handlers, running on Python 3.8+ and BS4, though development on it is slow enough that it's a riskier long-term bet than the others.

html2md is built for bulk, async processing, with automatic YAML frontmatter generation baked in, which fits naturally with migrating a site into Hugo or Jekyll. Fast, async-native, pulls metadata, ships a CLI. Needs Python 3.10+ and aiohttp.

As a shortcut: speed points toward trafilatura or html2md, customization points toward markdownify, type safety points toward html-to-markdown, and dependency-free simplicity points toward html2text. All six handle the basic job fine. The differences show up at the edges, in messy real-world HTML, in tables, in metadata, and in what breaks once volume climbs into the thousands of pages.

When a specialized small language model handles conversion better than a library

Rule-based converters hit a ceiling eventually, and that ceiling is the internet's sheer inconsistency. After the reference implementation mentioned earlier shipped in April 2024, user feedback drove a cycle of regex patches and heuristic tweaks that got harder to maintain with each addition and didn't generalize across languages. Patching regex forever isn't a strategy, it's a treadmill, and that's the pattern that pushes teams toward a model-based approach instead.

The insight underneath it is simple once it's said out loud: HTML-to-Markdown conversion is mostly a copy task, not a generative one. The model isn't inventing new text, it's deciding what to keep and what to drop. That's exactly the kind of task where a small, purpose-built model can match or beat a much bigger general-purpose LLM, because it doesn't need broad world knowledge. It needs narrow judgment applied the same way every time.

A first-generation pair of models built for this task shipped at 494 million and 1.54 billion parameters, taking raw HTML directly as input with no prefix instruction needed, and became available through Azure Marketplace and AWS SageMaker under a non-commercial license.

A second-generation version released in early 2025 improved on this in a real way. Built on a 1.5-billion-parameter base fine-tuned from Qwen2.5, it processes documents up to 512,000 tokens, which fixes a real failure mode in general-purpose LLMs: they tend to lose content buried in the middle or end of very long documents. On curated benchmarks, particularly for documents over 100,000 tokens, it beat GPT-4o-2024-08-06 and other larger models by 15 to 20 percent. A small model beating a much bigger one on the exact task it was built for is the whole argument for specialization, in one data point. It trained on two objectives at once: instructed Markdown extraction (converting HTML to Markdown while stripping navigation and ads, with support for custom extraction instructions) and schema-guided JSON extraction for structured output. The pipeline behind it used a three-stage draft-refine-critique data synthesis process combined with continuous pretraining, supervised fine-tuning, direct preference optimization, and self-play iterative tuning. The model sits publicly on Hugging Face.

Two problems in particular push teams toward this approach over a library. Real-world HTML is genuinely messy: legacy markup, embedded JavaScript, stray comments everywhere. A rule-based library applies fixed rules no matter what, while a model handles variation it's never seen before. Multilingual content is the other one. Regex-based heuristics degrade fast on non-Latin scripts, while a model trained across languages handles them natively, without a separate rule set for each one.

None of this comes free, though. Running a small language model locally at production speed needs a real GPU, something in the RTX 3090 or 4090 class at minimum. The cost doesn't disappear, it just moves from API tokens to compute, and any team pretending otherwise is just putting off the bill.

How conversion quality propagates into RAG retrieval accuracy

Retrieval-augmented generation has become the standard way companies deploy LLMs against their own data, precisely because it pulls information in at runtime instead of demanding a full retraining cycle every time the underlying content changes. That also means a RAG system can reach past whatever date its base model's knowledge cuts off at.

RAG's real job is grounding. It hands a model verifiable references tied to specific retrieved sources, which cuts down on hallucination and lets a team keep answers inside a defined knowledge domain instead of whatever the base model happens to know. None of that works if the retrieved content is garbled, and teams that treat conversion as an afterthought are quietly capping their own retrieval accuracy before a single embedding gets computed.

Bad conversion breaks retrieval in a few specific ways. Flattened headings mean a chunker has nothing to split on, so a single chunk can end up spanning several unrelated topics stitched together. Boilerplate that repeats across every page, the same nav bar, the same footer, inflates embedding similarity between chunks that have nothing to do with each other, which drags down ranking quality across the board. And lost code formatting means a code sample gets retrieved and handed to the model as ordinary prose, which routinely leads to broken code coming out the other side.

There's a freshness problem sitting underneath all of this too. A RAG system is only as good as how current and how clean its indexed content actually is. A pipeline that converts a page once and caches the result forever ends up reasoning over a frozen snapshot, not the live web that page belongs to. Real-time retrieval, where a system pulls the most current version of a source instead of leaning on a stale index, is turning into the expectation rather than the exception. That pushes conversion out of the realm of a one-time script and into something that has to run continuously, day after day.

Managed APIs versus self-built pipelines for production HTML-to-Markdown at scale

The three-stage pipeline described earlier isn't a mystery. Clean, convert, structure: any team can build it in an afternoon. Running it reliably across thousands of URLs a day is a different problem entirely, and it's the one that actually decides whether build-it-yourself holds up.

JavaScript rendering at scale means running a fleet of headless browsers, not a script that fires off one. Rate limiting and bot detection mean requests get blocked, throttled, or served fake content, and a pipeline has to catch and handle all three. Proxy rotation turns into a full-time operational job once volume climbs past a handful of domains. And content change detection, knowing when a previously converted page needs a re-fetch because the source changed, adds a whole layer of state tracking a one-shot script never had to think about.

None of that is exotic engineering. It's just engineering that has nothing to do with the actual value a team is trying to create, which is usually a better product built on clean retrieval, not a better web scraper. Maintaining a browser fleet, proxy rotation, and a change-detection layer forever consumes engineering hours that most teams would rather spend elsewhere. That's the calculation pushing more teams toward managed infrastructure for this layer instead of building it in-house: the three-stage pipeline is easy to stand up once, and genuinely hard to keep running well, at scale, forever.