Scraping Dynamic Pagination and Infinite Scroll for Data Pipelines

Scraping a paginated website used to mean one thing: bump a number in the URL, grab the HTML, repeat. That still works on a portion of the sites out there. On the other two-thirds, content shows up after a scroll event, a click, or a timer, and the HTML you get on first load is basically an empty shell. Getting this wrong doesn't throw an error. It just hands your pipeline the first twenty rows and calls it a day, and nobody notices until a report downstream comes up short.

Running everything through a headless browser is the wrong default, and it needs to be said because so many teams do it anyway. Browser automation is slower, more expensive, and unnecessary for most of what you'll actually run into. The right approach is knowing which of three patterns you're looking at, then picking the cheapest technique that fits that specific pattern. Most teams skip straight to browser automation because it feels safer. That backwards instinct is the single most expensive habit in this workflow.

The three pagination patterns and what each one looks like in the network layer

Open DevTools, go to the Network tab, and watch what happens when you paginate. This one habit tells you almost everything you need to know before writing a line of scraper code.

Numbered pagination is the old-school kind. Page state lives in the URL, ?page=2, ?page_num=3, that sort of thing. The server renders the whole page fresh each time, no JavaScript required. The Network tab confirms it: a plain document request, not an XHR or fetch call. Termination is easy to spot too, since the "Next" button either disappears or gets disabled once you hit the last page. This is the cheapest case by far. An HTTP client and an HTML parser get the job done, no browser required. Reaching for one anyway just burns compute for nothing.

Click-to-load pagination works differently. Pressing "Load More" fires a background request, and the URL in the address bar might not move at all. Check the Network tab and XHR or fetch calls show up, usually returning JSON or a chunk of HTML. There are two ways to handle this: replicate the API call directly, or automate a real click and read the DOM after it updates. Take the API route. It's faster and cheaper, full stop. Automation is the fallback, reserved for when the request parameters are signed or obfuscated in a way that's not worth reverse-engineering.

Infinite scroll looks the most different on the surface but runs on the same machinery underneath. Instead of a click, a scroll event or an IntersectionObserver fires the request. Scrapfly calls this "endless paging," a fair description, since there's no page count and no "last page" marker, just a stream that stops when it stops. A fast diagnostic: disable JavaScript and reload the page. If only the first batch of items shows up, everything after that is client-side rendered, and the choice is between intercepting the API or automating the scroll.

Reverse-engineering the hidden API: the most efficient path for both click-to-load and infinite scroll

Most "dynamic" pages aren't dynamic in the way they look. They're static templates pulling from a JSON API, and that API, not the rendered HTML, is where the real data lives. Skipping this step and jumping straight to browser automation is the single most common mistake in this whole workflow, costing the most compute for the least reason.

Finding the API is mechanical. Open DevTools, filter Network by XHR, trigger a scroll or a click, and look at what fires. Check the request URL, the headers, and the response body. Most of the time the response is structured JSON: an array of items, plus some kind of pagination pointer, whether that's an offset, a page number, or a cursor token.

Scrapfly's web-scraping.dev demo site makes this concrete. Scrolling on the testimonials page triggers a call to GET /api/testimonials?page=N. Once you know that, a browser isn't needed at all. Loop that request directly, increment the page parameter, and stop when the server returns a non-200 status. No rendering, no scroll timing, no waiting on JavaScript to finish executing. Just clean JSON, page after page, at a fraction of the time and compute cost of driving a browser.

The API route breaks down in three specific situations: the request carries a signed or dynamically generated token that can't be reproduced outside a real browser session, the endpoint checks for a session cookie tied to an authenticated browser, or the endpoint's shape changes often enough that maintaining a replica isn't worth the upkeep.

One gotcha trips people up constantly: the Referer header. A lot of API endpoints check that the request came from the parent page. Skip that header in a replicated call and a 403 shows up for no obvious reason, then an hour gets burned assuming the endpoint itself is broken. Copy the header over. It's a small detail that saves real debugging time.

When to use headless browser automation and how to drive it efficiently

Automation is the fallback, not the default, once the API path is closed off. That point deserves repeating, because so many pipelines reach for Playwright first out of habit rather than necessity. Playwright, Puppeteer, and Selenium can all simulate scrolling and clicking. Playwright has become the standard choice for new pipelines, mostly because of how cleanly it handles waiting and network state.

A typical scroll loop: navigate to the page, record the scroll height, then run window.scrollTo(0, document.body.scrollHeight) through page.evaluate(). Wait for new content, either with a fixed timeout or a wait-for-selector call, then compare the new scroll height against the old one. If they match, nothing new loaded, and the loop is done. Cap it at a maximum number of iterations regardless, because a feed that claims to be infinite might genuinely be close to it, and a pipeline job that never returns is its own kind of failure.

Click-to-load automation follows a similar shape, just with a button instead of a scroll trigger. Find the "Load More" element by selector or aria-label, confirm it's enabled, click it, wait for the network to settle, repeat. Exit when the button disappears or gets disabled.

Headless mode is the default for pipeline work, since nothing needs to render visually. Some sites run anti-bot detection that specifically fingerprints headless browsers, though, so stealth plugins or a full browser profile become necessary there. For teams that don't want to run and maintain browser infrastructure themselves, managed scraping APIs with flags like render_js=True and auto_scroll=True abstract the scroll loop entirely. Worth knowing about even for teams that build the loop by hand most of the time.

Treating pagination state as a graph to prevent loops, duplicates, and silent data loss

A linear model of pagination, page 1, then page 2, then page 3, breaks down fast in production. "Next" links skip pages. Feeds reorder themselves between requests. Cursors expire or repeat. ScrapingAnt's analysis is blunt about it: treating pagination as a graph is foundational for scraping meant to run reliably in 2025 and beyond. A linear mental model is the wrong one to hold onto, being the assumption that breaks first once a site changes anything upstream.

Here's the abstraction. A node is a unique pagination state, defined by the URL and its query parameters, a cursor token, or a hash of the items retrieved. An edge is the action that moves from one node to another, a click, a scroll trigger, a filter change. A loop happens when an edge leads back to a node already visited, and missing that means the scraper happily re-fetches the same data forever.

How a node gets keyed depends on the pagination type. Numeric pages get tracked by visited page number per filter and sort combination. Cursor-based APIs need the cursor token tracked together with its associated parameters, since the same cursor value under a different filter is a genuinely different node. Infinite scroll gets keyed by hashing the request parameters, or hashing the content of the item IDs in the batch.

Before issuing a new request, build the node key and check it against a visited set. If it's already there, stop that path. This is depth-first search with cycle detection, though in practice most pagination graphs collapse down to a single path, so the overhead stays minor.

A subtler failure mode: cursor tokens that differ but point to identical or overlapping content. This shows up constantly in personalized or rapidly updating feeds, where the token changes but half the items are ones already scraped. Key-only deduplication won't catch it. A content hash will. Compute a hash from the sorted item IDs on each page, keep a small cache of the last several hashes, and compare against it. ScrapingAnt flags this exact scenario as one of the main causes of duplicate data at scale, and it's easy to miss when the pagination key is the only thing being checked.

Set a hard cap on page or batch count no matter what. Any feed that claims to be infinite should still hit a wall on your terms, not its own. Log it when the cap gets hit, too. Silent truncation is worse than an error, because nobody downstream knows to go looking for the missing data.

Cursor-based and token-based APIs: the pagination pattern that looks different but follows the same graph rules

Cursor-based pagination is the standard in modern GraphQL APIs and shows up more and more in REST APIs too. Instead of ?page=N, the response carries a field like next_cursor, endCursor, or nextPageToken pointing to the next batch.

In GraphQL specifically, the pageInfo object carries endCursor alongside hasNextPage. When hasNextPage comes back false, the pagination is done. Clean, explicit, no guesswork required.

The node key here should combine the endpoint path, the cursor value, and any stable parameters like filters or sort order. Skip the parameters, and two genuinely different states risk getting treated as one node just because the cursor value happens to match.

Design around one thing specifically: cursor tokens are frequently opaque and time-sensitive. A cursor captured a few hours earlier might resolve to different data by the time it's used, or it might just error out. Build the pipeline to treat that as an expected condition, not a crash.

A large share of what looks like "infinite scroll" on the frontend is a cursor-based API underneath. Once the network calls get intercepted (see two sections up), this becomes obvious. At that point, the scraper should work directly off the API's cursor field rather than simulate scroll events, since the cursor is the actual source of truth and the scroll is just the UI's way of triggering it.

Termination conditions: how to know when pagination is genuinely finished

Every pagination pattern needs its own definition of "done." Getting this wrong is one of the most common ways pipelines quietly under-deliver, and it's usually invisible until someone downstream asks why the dataset looks thin.

For numbered pagination, a disabled or missing "Next" button is the signal. Fetching page N+1 and getting an empty results array, or a redirect back to page 1, means the same thing. For click-to-load, it's the "Load More" button going away, hidden or disabled. For infinite scroll driven by a scroll simulation, it's the scroll height staying flat after a wait, meaning nothing new got injected into the page. For API and cursor-based pagination, watch for hasNextPage: false, an empty next_cursor or nextPageToken field, an HTTP 404 or 400 on the next request, or an empty items array in an otherwise successful response.

A few cases are murkier and deserve specific handling. If a feed returns the exact same items as the previous batch, that's the content-hash check from earlier doing its job, catching the repeat before another wasted request goes out. If a feed returns HTTP 200 with an empty array, treat that as a genuine stop rather than something to retry. A 429 rate-limit response is not a termination signal at all: back off, wait, and retry with exponential delay, since giving up here just means missing data that was never actually finished.

Keep a hard iteration cap in place regardless of what signal is being watched for. A misconfigured site that never sends a real stop signal shouldn't be able to run a pipeline indefinitely. And whichever way the loop ends, log the reason explicitly, proper termination or cap kicking in. A pipeline that truncates silently is failing quietly, and quiet failures cost the most time to catch later.

Turning paginated web data into clean, pipeline-ready output for AI systems

Raw HTML, even once fully paginated through, isn't something to feed an LLM directly. Language models are trained on linear text, while HTML is tree-structured markup, and that gap has to close before the data is actually useful.

Markdown is the common middle ground, and it should be the default output format for most pipelines, not JSON. Converting extracted content into Markdown can cut token usage by up to 70% compared to raw HTML, according to SearchCans, and one community benchmark found Markdown roughly 16% more token-efficient than JSON for retrieval and summarization work specifically. That's a real cost difference at scale, especially across a pipeline pulling thousands of paginated records a day.

JSON still wins in one clear scenario: when downstream code needs named fields and structure matters more than squeezing out tokens. That's where schema-driven extraction comes in. Define a schema up front for exactly the fields a page needs to produce, product title, price, review count, URL, and apply it consistently across every page in the pagination loop. Two benefits fall out of this. Output stays uniform no matter how much the underlying HTML varies from page to page, and missing fields get caught one page at a time, rather than surfacing as a gap after the entire crawl has already finished.

There's a freshness angle worth taking seriously too. On datasets limited to knowledge available after February 2025, baseline LLM performance without external retrieval drops to a score of 0.2022. The gap between systems using retrieval-augmented generation with live web corpora and those that don't widens from 16.3% on standard validation data to 44.16% on post-cutoff test data. Paginated live web scraping is one of the main ways fresh data actually gets into a RAG system in the first place.

One last step before anything goes into a vector store: deduplicate on item IDs. Paginated scrapes overlap more often than expected, especially on feeds that reorder or personalize, and indexing the same record twice just burns embedding compute for nothing.

Operational patterns for running paginated scrapers as reliable pipeline components

A scraper that runs once and works is a script. A scraper that runs on a schedule, handles failure gracefully, and reports what it did is a pipeline component. That difference matters more than it sounds. Treating the two as interchangeable is how "working" scrapers turn into silent liabilities six months later, usually right around the time someone upstream changes a template and nobody notices for a week.

Set a crawl cadence that matches how often the underlying data actually changes: daily for fast-moving listings, weekly for slower catalogs. Diff each run's output against the one before it. That diff is often more valuable than the raw scrape itself, since it shows exactly what changed instead of forcing someone to eyeball two full datasets side by side.

Everything covered above, the graph-based state tracking, the content-hash deduplication, the explicit termination logging, the hard iteration caps, exists to support one operational goal. It was about handling many pages and many sites cleverly. It's about the whole system holding up, so that when the scraper runs unattended at 3 a.m. for the two-hundredth time, it stops for the right reason, reports what happened, and hands the pipeline clean data instead of a partial batch that looks complete but isn't.