12

For most internet users, the public web, accessible by search engines like Google, is the beginning and the end of the Internet. More savvy users understand that there are large numbers of sites that are not accessible by Google, and are accessible only through browsers like Tor. Collectively, this is called the Deep Web.

Indeed, in Season 2 of House of Cards, the Deep Web even makes an appearance, and the screen clearly shows the Tor Browser being used.

A paper at the University of Michigan specifically states the deep web is orders of magnitude larger than the surface web. To wit:

Public information on the deep Web is currently 400 to 550 times larger than the commonly defined World Wide Web. The deep Web contains 7,500 terabytes of information compared to nineteen terabytes of information in the surface Web.

Here's my issue - when I've tried out Tor, the lack of a search engine that actually returns results makes me wonder how they gathered that data. So, in addition to testing the veracity of the claim that the Deep Web is orders of magnitude larger than the surface web, how do researchers actually prove this?

And how can I get something other than a broken link if I want to see for myself?

Improve this question
4
  • @jwenting - please avoid pseudo-answers in comments. You made a lot of claims in that comment without any refs
    – Jamiec ♦
    Feb 19, 2014 at 11:23
  • 2
    Your definition of Deep Web (not accessible through browsers) does not match the cited link (roughly, not accessible through Google Search).
    – Oddthinking ♦
    Feb 19, 2014 at 13:15
  • 2
    Web is by definition accessed by web browser, deep web is accessed by web browser, but for some reason unindexed. In 2001 hardly anything dynamic, be it server-side or JavaScript, was indexed. Nowadays it is indexed. For example by 2001's definition anything on Amazon.com (with exception of front page), would be considered Deep Web.
    – vartec
    Feb 19, 2014 at 19:45
  • See en.wikipedia.org/wiki/…, cyberwarzone.com/best-tor-search-engines, wired.com/2014/04/grams-search-engine-dark-web, and deepweblinks.org for links to various tor search engines. Simple google search for "tor search engine".
    – ike
    Oct 21, 2014 at 19:21

1 Answer 1

Reset to default
14

The paper you link to is referring to 'Deep Web' meaning unindexed data, and it's claims may be roughly correct in this sense - at the time it was published. It's not referring to the Tor based network of hidden sites you also mention, however. The term DarkNet is often used for these sites, so for clarity, I will refer to 'Deep Web' when discussing normal web content that is unindexed, and DarkNet when referring to encrypted, hidden, anonymised websites.

Deep Web as unindexed pages

The paper you refer to is actually from 2001, and also seems to be a founding basis for the Wikipedia page on Deep Web. Because the paper is so old now, the likelihood of it's estimates being still correct, or even of the proportions of information being correct is in my opinion, remote. Opinions aside, the paper is very out of date, and a more up to date reference would be needed to estimate the size of indexed vs non indexed data on the web.

Darknets

Anonymised proxy delivered websites, like The Silk Road and it's ilk, are the sites delivered over Tor and similar relays using obscure urls like https://silkroad3dff81.onion (deliberately not the actual Silk Road URL, although even the real one would be useless now it has been closed). These are more correctly known as DarkNets, or Dark Internet, although the terminology gets a bit mixed up.

Size of Darknet

I've looked through as much as I possibly can in the time I have on how big the DarkNet actually is, that is to say, how much data on the internet is stored in DarkNets. The search has been rather problematic, as most articles that are most prominent on the internet are actually quoting the 7.5TB or a reduction of this to a percentage, multiple etc. and this came from the orignal 2001 paper, you refer to, which can't be referring to Tor based Darknets, as it was published one year before Tor was launched.

The only practical way to measure the DarkNet, because the information is encrypted whilst passed, is by network analysis, which will only ever be a rough estimate (much encrypted data will not be DarkNet traffic). This IEEE paper covers an approach to measuring the DarkNet, but does not offer an actual figure for either bandwidth usage, or data stored.

The inherent problem of this question is that you are asking the size of hidden and secret data. To really size it, you'd need to be able to see it all, and even if you could, you couldn't know that there wasn't more information you were unable to see. So the answer is that the estimates given by journalist constantly are probably wrong, the size is probably changing all the time, and the size of the DarkNet is basically indeterminate.

Also of interest is this paper, which refers to the technical plausibility of growth of the DarkNet, concluding that growth is very possible, and likely.

Improve this answer
5
  • You are using a few terms incorrectly in this post. From the Wikipedia article on Dark Internet, "Whereas deep web and darknet stand for hard-to-find websites and secretive networks that sometimes span across the Internet, the dark Internet is any portion of the Internet that can no longer be accessed through conventional means." Also, Darknets are a small subset of the deep web. The two are not mutually exclusive. And in addition to this a Darknet is not a singular thing, but is a network. There are many Darknets.
    – Cruril
    Feb 19, 2014 at 17:07
  • And in addition to my previous comment, TOR falls under the Deep Web, not a Darknet. Darknets are more used for file sharing or for decentralized F2F (friend-to-friend) networks. A F2F network is a private P2P (pier-to-pier) network.
    – Cruril
    Feb 19, 2014 at 17:12
  • 2
    I clarify what I mean by each term at the start of the answer, and also mention that the terms are sometimes used interchangably. I've used them as I normally do in this, but they do get mixed up, and which is 'correct' is basically down to which usage becomes most popular. You're right about DarkNets being things, rather than a thing, so to clarify, when I refer to DarkNet in the singular, I'm talking about the multitude of DarkNets, rather than a specific one.
    – Owen C. Jones
    Feb 19, 2014 at 17:15
  • I suspect that tor is mostly used to anonymize access to perfectly ordinary websites. Just ones you don't want your employer/spouse/ISP/government known you have accessed. Compare and contrast with, say Freenet, which is a different type of content network from the web, operating over a different protocol.
    – dmckee --- ex-moderator kitten
    Feb 19, 2014 at 20:53
  • Tor, in that sense, is used to mainly achieve anonymity for users, yes. A by product of the way TOR works was that nodes on the network have a hash id, something like abuhdaksdh232k34.onion, which the TOR browser can access directly. Host a site on a server using a TOR node, and your site can be accessed through a .onion address. It's possibly not the most popular method of creating dark sites, but it is one, it was used by SilkRoad, Black Market Rebooted and a fair few others.
    – Owen C. Jones
    Feb 20, 2014 at 18:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .