The deep web is the part of the World Wide Web that search engines do not index, such as content behind logins, paywalls, forms and private databases.
What the deep web includes
The term was popularised by Michael K. Bergman in a 2001 paper describing how much online content was invisible to search engines. Most of it is ordinary and legitimate:
- webmail inboxes, online banking and cloud storage accounts;
- company intranets, admin panels and internal wikis;
- medical records, academic databases and paywalled news;
- dynamic pages generated only in response to a search form.
Pages may also be excluded deliberately through robots.txt or “noindex” tags. The deep web is much larger than the indexed “surface web”.
Deep web vs dark web
The two terms are often confused. The dark web is a small subset of the deep web that is intentionally hidden and reachable only with special software such as Tor. Using the deep web is something nearly every internet user does daily – checking email is enough – while the dark web is associated with anonymity and, often, criminal marketplaces.
Why the deep web matters for security
Because deep web resources hold sensitive data, they are a key target. Leaked or reused credentials give attackers access to email and banking accounts, exposing personally identifiable information. A common mistake is assuming that unindexed means private: misconfigured cloud storage buckets, Elasticsearch databases and admin panels are regularly found by scanners and services such as Shodan, even without links from public pages. Researchers and attackers alike use open source intelligence techniques to discover such exposed resources.
Best practices
- Protect every non-public resource with strong authentication and MFA, not just obscurity.
- Do not rely on robots.txt or unguessable URLs to hide sensitive data.
- Regularly scan your own external attack surface for exposed databases and panels.
