Ethical and Legal Boundaries of Web Scraping for AI

Agentic AI traffic jumped 7,851% in 2025. By the fourth quarter, TollBit's data showed 1 in every 50 website visits came from an AI scraper, up from 1 in 200 just months earlier. That's not a rounding error. That's a structural shift in how the web gets read, and it means scraping for AI has stopped being a niche engineering problem that legal teams can safely ignore later.

Most developers start with a simple intuition: if the data is public, it's fair game. That intuition is wrong, or at least incomplete, and the rest of this piece exists to unpack why. There's no single law that governs web scraping for AI. Instead, there's a stack of overlapping regimes: terms of service, copyright, privacy law, technical signals like robots.txt, and now regulation specific to this technology in the EU. Each layer creates its own exposure, and compliance with one doesn't buy you a pass on the others. Treating this as a framework, rather than a list of separate concerns, is the only way to build a pipeline that doesn't need to be rebuilt six months after launch.

What the foundational case law actually settled, and what it left open

Start with hiQ Labs v. LinkedIn, because it's the case everyone cites and the one most people over-read. An appellate court's 2022 ruling on remand said that scraping publicly accessible data, without bypassing a login wall, likely raises a serious question about whether it counts as "unauthorized access" under the relevant federal computer access law. Note the hedges in that sentence: "likely," "serious question," "raises." This was a preliminary injunction standard, not a final merits ruling. It's a strong signal, not a settled rule.

Here's what hiQ did not touch. It said nothing about whether scraping in violation of a site's terms of service amounts to a breach of contract, which is a separate civil claim entirely. It said nothing about copyright, meaning what you're allowed to do with data once you've collected it. And it has zero bearing on privacy law, which runs on its own track regardless of how the CFAA gets interpreted.

Meta v. Bright Data, decided in January 2024 by Judge Edward Chen in the Northern District of California, sharpens this distinction. Bright Data won summary judgment because Meta's terms of service didn't bind someone who wasn't logged in. Scraping while logged out meant Bright Data wasn't a "user" subject to those terms in the first place. But the same ruling confirmed the flip side: scraping behind a login, where contractual terms do apply, can absolutely constitute breach, even when the underlying content is otherwise visible to the public. The access method matters as much as the content itself.

Van Buren v. United States, the Supreme Court's 2021 decision, narrows the CFAA further. "Exceeds authorized access" means reaching parts of a system that are off-limits, not misusing access you already have for a purpose the terms of service don't like. Put these three cases together and a pattern emerges: access-method legality and data-use legality are two different questions, answered by two different bodies of law. You can win on CFAA grounds and still get hit with a copyright claim, a contract claim, or a privacy claim on the exact same scraping conduct.

Scale creates its own risk, separate from access rights. In related litigation, trespass-to-chattels claims against scrapers have struggled where platforms cannot show actual server impairment. Scraping, without proof that it degraded performance, has been treated as no more burdensome than a person browsing with a web browser. No harm shown, no claim.

Jurisdiction is also less predictable than developers assume. Ryanair v. A travel-booking company showed that similar legal claims can reach international companies even when neither party is based in the United States. Assuming a scraper is safe because it operates outside one country's borders doesn't hold up.

The floor is what case law gives you here, not a ceiling. Clearing the CFAA bar still leaves copyright, contract, and privacy exposure fully intact.

Terms of service as the first practical compliance layer

Violating a site's terms of service isn't a crime, but it carries three real consequences. The platform can cut off access immediately, without warning. It becomes the basis for a civil breach-of-contract suit. And in front of a judge, it reads as bad faith, which can quietly undercut an otherwise solid defense on a separate claim.

Reddit's lawsuit against Anthropic, filed in June 2025 in California and later remanded to California state court, lays this out concretely. Reddit's User Agreement applies to every visitor, bots included, and bars commercial exploitation of the platform's content. Anthropic allegedly declined to license Reddit's "Compliance API," kept scraping anyway, and bypassed both robots.txt and IP rate limits in the process. The claims are breach of contract, unjust enrichment, trespass to chattels, tortious interference, and unfair competition. Notably, there's no copyright claim in that suit at all. It's a contract case, built entirely on the terms-of-service layer.

Reddit's separate suit against Perplexity AI, alongside three scraping and proxy providers, adds a wrinkle: routing around a block doesn't make the block disappear. The providers allegedly pulled Reddit content out of Google search results instead of scraping Reddit directly, and Perplexity allegedly obtained Reddit content through at least one of them. Going through a third party doesn't dissolve the exposure. If the underlying data was fenced off, buying it secondhand doesn't unfence it.

The practical shift here: platforms increasingly offer a licensed path. Reddit's Compliance API is one example. When that path exists and a company scrapes around it instead, courts and regulators are far less likely to treat it as a gray area. It starts to look like willful circumvention.

Before scraping anything, a real review means checking a handful of things. Does the site explicitly ban automated access or commercial reuse of extracted data? Is the access happening logged-in or logged-out, since Meta v. Bright Data made that distinction legally meaningful? Does a licensing or API pathway already exist, because its existence tilts the equities toward using it? And is any of this written down, since a documented review is worth a lot if a dispute ever lands in front of a judge.

None of this is a one-time check, either. Terms of service change, sometimes without much notice, and a pipeline that was clean at launch can drift out of compliance without anyone touching the code. Monitoring for those changes is operational work, not a box to tick once.

Copyright and AI training: where the litigation frontier is moving

The training-data lawsuits are a different animal from the access cases above. They're not asking whether collection was authorized. They're asking what happens to the content after it's inside the pipeline.

Bartz v. Anthropic, decided by Judge Alsup in June 2025, found that training a language model on lawfully obtained books is "exceedingly transformative" fair use. That sounds like a green light, but the ruling explicitly carved out books obtained through piracy. It's a narrow, fact-bound win tied to lawful acquisition, not a blanket doctrine that covers any training use.

Kadrey v. Meta, decided by Judge Chhabria the same month, cuts the other way in tone even though the plaintiffs lost. Chhabria's opinion signaled that LLM training on copyrighted works could be found infringing, leaving the door open for future plaintiffs in stronger factual postures. Read that as a roadmap for the next round of plaintiffs, not as broad legal permission for AI training on copyrighted work.

Reddit v. Perplexity brings in a different legal theory entirely: DMCA Section 1201, which covers anti-circumvention of technical protection measures. That's distinct from the "was this public data" question at the heart of hiQ. It targets the act of bypassing a technical barrier, full stop.

International courts are getting involved too. A group of Canadian news companies filed suit against OpenAI in the Ontario Superior Court of Justice in November 2024, the first major case of its kind in Canada. It follows an earlier consent order, Toronto Real Estate Board v. Mongohouse.com in 2019, which resulted in a consent order addressing unlawful scraping of MLS database content. The OECD's 2025 report on intellectual property and AI training adds another data point: unlicensed use of scraped content can breach copyright even when the data looks public on its face.

For anyone building a pipeline, a few things follow directly. Provenance matters: data that was lawfully accessed is not automatically cleared for training use. Fair use and fair dealing arguments are jurisdiction-specific and outcome-uncertain, which makes them a courtroom defense, not something to build a compliance program around. Source copies matter too, per the Bartz carve-out: if pirated material entered the pipeline anywhere upstream, a transformative-use argument may not save it. And structured facts, prices, listings, rankings, sit in a different category since facts themselves aren't copyrightable, though EU database rights complicate that picture.

Clearview AI is the clearest illustration of how these layers stack. A facial-recognition company settled a class action in one country in 2025. for roughly $51 million plus a 23% equity stake for the plaintiffs, on top of separate enforcement actions on another continent. The photos Clearview scraped were public. That didn't matter, because the data type, biometric identifiers, triggered a completely separate legal exposure. Winning the access-method argument protected nothing once the data-type problem showed up.

Privacy regulations: GDPR, CCPA, and the lines they draw for scraped data

Start with the boundary that matters most: non-personal data like prices, product listings, or rankings sits entirely outside GDPR. Facts aren't creative works and they aren't personal data, so privacy law only switches on when the scraped data relates to an identifiable person.

Once it does switch on, GDPR reaches further than most developers expect. It applies to data about EU residents no matter where the company doing the scraping is located. "We're not a European company" isn't a defense. Penalties run up to €20 million or 4% of global annual revenue, whichever is higher, and total GDPR fines have crossed €4 billion since the regulation took effect.

Enforcement has also changed character. It used to be reactive, mostly complaint-driven. Now regulators go looking. European data protection authorities, including France's CNIL, have pursued scraping-related investigations in 2024 and 2025, with enforcement actions resulting in notable fines. CNIL fined a company called KASPR €240,000 for pulling LinkedIn data without proper consent, a useful reminder that professional data visible on a public profile is still personal data once it identifies someone. Regulators and data protection guidance have increasingly treated robots.txt compliance as a factor in Legitimate Interest assessments: ignoring a Disallow directive can read as a negative signal in an investigation, which ties this section directly into the robots.txt discussion below.

CCPA and its update, CPRA, cover California residents regardless of where the scraper sits. Under these rules, a company can't scrape and sell personal information without offering an opt-out. Deletion and access requests have to be honored. Fines run $2,500 per unintentional violation and $7,500 per intentional one, and intentional violations draw the higher penalty.

Buying scraped data from a vendor doesn't transfer the compliance burden either. Processing personal data purchased from a third party creates its own compliance obligations for the buyer, and a contractual arrangement with the seller does not eliminate that exposure. B2B contact data is a particular trap here: professional data that identifies an individual can qualify as personal data under GDPR even when aggregated and resold as business intelligence. The "it's business data, not personal data" defense hasn't held up in enforcement actions.

A working compliance posture, in practice, means running a Legitimate Interest Assessment before processing any personal data pulled from public sources, stripping PII at collection time unless it's specifically needed, building deletion-request handling into the pipeline from day one rather than bolting it on later, and keeping processing logs and data provenance records on hand, since regulators ask for exactly these documents during investigations.

How robots.txt evolved from a voluntary standard into a legal instrument

Martijn Koster proposed the Robots Exclusion Protocol in 1994 as a purely voluntary, honor-system standard. No enforcement mechanism existed, and it carried no legal weight whatsoever. Three decades on, courts, regulators, and now lawmakers treat robots.txt compliance as evidence, and sometimes as a deciding factor, in whether scraping was done in good faith.

The EU AI Act turns that evidentiary weight into an actual penalty. Obligations under the Act have applied since August 2, 2025, and starting August 2, 2026, an AI company that ignores robots.txt risks a fine of up to €15 million or 3% of worldwide annual turnover, whichever is higher. Article 53, already in force, requires providers of AI models to put a copyright compliance policy in place and to deploy what the Act calls "state-of-the-art technologies" for detecting rights reservations.

A court in one European country has already put teeth into the machine-readability requirement. The Hamburg Higher Regional Court ruled on December 10, 2025, in Kneschke v. LAION, that a text-and-data-mining opt-out only counts if it's machine-readable. A plain-language notice buried in a site's terms of use isn't enough. The reservation has to be implemented in a form a machine can actually read and act on, whether that's robots.txt, an X-Robots-Tag HTTP header, or the TDM Reservation Protocol, though the court stopped short of naming a closed, exhaustive list.

Industry codes are catching up too. The GPAI Code of Practice, from July 2025, requires signatories to run crawlers that read and respect robots.txt under RFC 9309. Yet the gap between rule and practice is still wide: AI crawlers averaged 156 violation requests per site over a three-week stretch in 2025, even as the legal stakes for ignoring the standard kept rising.

There's a real enforcement headache buried in all this, too. A robots.txt rule is addressed to a specific named user agent, and the crawler gets to declare its own name. That means disputes often come down to arguing over which log entries belong to which operator. The major training crawlers are at least publicly named: GPTBot for OpenAI, ClaudeBot for Anthropic, CCBot for Common Crawl, and Google-Extended, the robots.txt token covering Gemini and Vertex AI training rather than a separate fetching crawler.

Emerging industry conventions for signaling AI crawling preferences push this even further by letting site owners grant or deny access based on intended use, search versus training, rather than just crawler identity. And Cloudflare's Pay Per Crawl infrastructure, which launched in private beta on July 1, 2025, signals something bigger: access negotiation between crawlers and publishers is turning into actual infrastructure, not just a policy statement in a text file. Robots.txt compliance is quickly becoming a floor rather than a finish line. A crawler can respect it fully and still need to answer a harder question: what is this data being collected for?

The EU AI Act as a scraping-specific compliance layer on top of GDPR

The AI Act doesn't replace GDPR. It sits on top of it, which means a team that's already done its GDPR homework still has a separate set of AI Act obligations to work through.

Since August 2025, providers of general-purpose AI models have been under transparency requirements: publish a summary of training content, and maintain a copyright compliance policy. Then, starting August 2, 2026, fines for ignoring robots.txt obligations take effect for AI companies, with fines reaching up to 7% of global annual turnover for the most serious violations, compared with GDPR's ceiling of 4% of global revenue.

What makes the AI Act structurally different is that it's the first regulation to directly target what content an AI system trains on, rather than only regulating how personal data gets processed. That's a real shift in the shape of the compliance landscape, not just an incremental add-on. The Article 53 copyright compliance requirement, with its call for "state-of-the-art technologies" to detect rights reservations, effectively imports the Hamburg court's machine-readability standard into binding EU law, tying robots.txt and its successors directly to statutory obligation rather than leaving them as a courtesy.

Put the whole framework together and the message is consistent across every layer: a scraper that clears one hurdle, CFAA, terms of service, copyright, GDPR, robots.txt, has not cleared the course. Each layer runs on its own logic and its own enforcement body, and each one can catch what the others miss.