Web scraping is the automated extraction of data from websites or apps by bots that download pages or call interfaces and save the content in structured form.
How web scraping works
A scraper requests pages like a browser, parses the HTML and extracts the desired fields – prices, product descriptions, profiles, contact details or comments. Modern scrapers use headless browsers to execute JavaScript, rotate IP addresses through residential proxy networks and imitate human behaviour to avoid blocking. Many also call a site’s API directly, including undocumented mobile app endpoints.
Legitimate and malicious use
Scraping powers search engines, price comparison sites, academic research, archiving and OSINT. Problems arise when bots ignore terms of service, overload servers or collect personal information:
- In 2021, data of about 533 million Facebook users was published online; Meta said it had been scraped in 2019 by abusing a contact import feature.
- Clearview AI built a facial recognition database from billions of images scraped from the web and was fined by several European data protection authorities.
- Competitors scrape prices and content, and fraudsters harvest email addresses and phone numbers for spam, phishing and targeted attacks.
The legal situation varies: in the US, the hiQ Labs v. LinkedIn case addressed whether scraping public data violates anti-hacking law, while in the EU scraped personal data remains subject to the GDPR.
How to protect against abusive scraping
- Rate-limit requests per IP, account and API key, and require authentication for sensitive data.
- Use bot management that analyses browser fingerprints and behaviour, with a CAPTCHA as a fallback challenge.
- Avoid exposing sequential IDs and bulk lookup features that make enumeration easy.
- Publish clear rules in robots.txt (RFC 9309) and terms of service, and monitor for unusual traffic.
