(Image: Istvan Szugyiczky)
Whether for privacy or profit, much of the internet is hidden away – we reveal the places that Google won’t take you
OVER the past two decades, we’ve created a whole new world that’s visible only through a screen. Visibility and transparency have long been hallmarks of the online world. We’re used to being able to see anything posted on the web, and we’re used to the idea that anything we do there may be visible to anyone else.
That’s not true though. To understand why, consider how you find things online. It’s tempting to think that anything anyone has ever published can be magicked to your screen with the right keywords. That’s partly true – if Google or Bing has indexed it, and your internet service provider (ISP) lets you go there, you’ll find it. But those “ifs” are bigger than they may seem. It turns out that an even greater proportion of the online world is dark than of the physical universe: it is estimated that just 0.03 per cent of the web is searchable.
Like dark matter and dark energy, we know the dark web is out there, but we can’t see it directly. Entire uncharted realms lie beyond the reach of the crawlers that tirelessly catalogue the web on behalf of search giants: databases, internet relay chat records, and the raw data behind research papers. Much of it is unsearchable, because most people aren’t searching for it. And if they do search for it, and don’t find it, they assume it’s not there to be found. “There’s this underlying assumption that if it’s not online it doesn’t exist,” says Ken…
