Depends on the interpretation of what handle and process five zettabytes of data really means:
- having 5ZB readily available for online processing?
- having 5ZB in off-line archive, while having only meta-data readily available for on-line processing?
- having 5ZB incoming traffic, discharging most of it, storing and processing only interesting stuff?
If you're going to build something as big as NSA datacenter, instead of buying readily available hardware, you might go for custom builds using big data industry standards.
Facebook's Open Compute Storage specifies Open Vault storage, which is designed to house 30 3.5" HDDs in 2U case. Which with currently available HDDs would mean density of 120TB per 2U.
You can put 21 of these in one standard 42U high 19" rack, meaning you'd have 2520TB per rack.
Each rack having a base of about 0.62m². Adding some very minimum clearance, 1m² per rack would be the very lowball estimate.
That's assuming that you'd be willing to have machines custom build for your datacenter. Otherwise it's off the shelf hardware. For example Dell PowerVault MD1200 configured with 12 4TB SATA drives costs $11,254.21; That of course is retail price, which bulk orders you get hefty discounts. Similarly, you could get a cloud server such as PowerEdge C6145, which can also be equipped with 12 4TB SATA drives. They don't provide they pricing on-line, but if I remember correctly, that kind of setup it'd be around $15-20K.
Both of above give you density of 48TB per 2U. Which means 1004TB per rack.
NSA Utah datacenter is reported to have 10,000m² dedicated to servers (100,000m² including administrative and support buildings). So really, really high ball estimate would be that they could theoretically squeeze 25EB of cloud storage in there.
That's only 0.5% of 5ZB.
If you're considering offline archiving of data, you might use tape libraries. It's not typically used in Big Data, but then again, NSA isn't typical Big Data company.
Problem is, that even with highest density IBM System Storage TS3500 Tape Library, you can store 180PB per library, which has footprint of at least 15m². Which means about 420,000m² would be needed just for the tape libraries. Whole NSA Utah complex is estimated to have "1 million or 1.5 million square feet" (93,000m²-149,000m²), but it's also been said that it's "100,000 square feet (9,300m²) of data center space and greater than 900,000 square feet (83,600m²) of technical support and administrative space" (source). Purpose of the buildings might be kept secret or intentionally misleading, external dimensions cannot. So clearly there is no way to squeeze 5ZB of tape archive there, even if buildings said to be administrative are actually purposed for tape archive.
So let's assume that even if you'd manage to somehow archive 5ZB of raw data, as I've calculated above even if you'd use all your datacenter space for on-line processing, that would mean you're storing only 5KB of metadata for each 1MB of data. While for media files that seems more than enough, I really doubt you can achieve ratio anywhere near that for emails, chats, text messages, tweets etc. Of course they might filter out information they've not interested in and not store any metadata related to that information. But in that case there is no point in archiving that information off-line, because you don't have path of access — you don't have metadata pointing to it, nor can you mine the data that is archived on tapes.
Assuming that it's 5ZB per year, that's a sustained throughput of 570PB per day, or 160TB per second. Leaving apart computing power needed to process that, question is how that data would be transported there? Current internet infrastructure isn't anywhere nearly ready for that kind of traffic. Three of the biggest internet exchange points are located in Europe having maximum throughput of 2.2TBps, 2.1TBps and 1.6TBps respectively. US is trailing way behind with top US only exchanges having only 0.28TBps and 0.25TBps, another one, Equinix has total of 1.4TBps across 12 countries on 4 continents. In other words it's completely infeasible to push 5ZB per year into a datacenter, especially in such a remote location. Nor, as you can see, necessary, even if you'd want to process all of Internet traffic.
- having 5ZB readily available for online processing?
- having 5ZB in off-line archive, while having only meta-data readily available for on-line processing? Not possible with current technology and sizes of the buildings as reported
- having 5ZB incoming traffic, discharging most of it, storing and processing only interesting stuff? Regardless of processing, it's just not possible to have that amount of incoming traffic
HighScalability.com has a guest post "PRISM: The Amazingly Low Cost Of Using BigData To Know More About You In Under A Minute" by BugSense Founder/CTO Jon Vlachogiannis and Head of Infrastructure at BugSense Panagiotis Papadomitsos.
They calculate for storage needed as 3.75EB.
Facebook: 500 TB/day * 30 = 1.5 PT/month (source)
Twitter: 8 TB/day * 30 = 240 TB/month 8 TB/day (source)
Email/Other info: 193PT/month Google says 24 PB per day (2008). Five
years later lets assume this is 8 times bigger = 192 PB. Now, real
user information is 1/3 = 64 PT/day (source)
Mobile traffic/machinetomachine exchanges/vehicles etc: 4000 TB per
day = 117 PB/month (source)
Total Data =~312PB month
The prices below correspond to renting offtheshelf servers from
commercial highend datacenters (considering the data will be stored
in a distributed filesystem architecture such as HDFS). This is a
worst case scenario that does not include potential discounts due to
renting such a high volume of hardware and traffic or acquiring the
aforementioned hardware (which incurs a higher initial investment but
lower recurring costs) . The hardware configuration used for
calculating costs in this case study is comprised of a 2U chassis,
dual Intel Hexacore processors, 16 GB of RAM, 30 TB of usable space
combined with hardwarelevel redundancy (RAID5).
We’ll be needing about 20K servers, put into 320 46U racks. Cost for
the server hardware is calculated to be about €7.5M / month (including
servers for auxiliary services). Cost for the racks, electricity and
traffic is calculated to be about €0.5M / month (including auxiliary
devices and networking equipment).
Total hardware cost per year for 3.75 EB of data storage: €168M
And comparison with Facebook:
It’s no secret that Facebook stores a lot of data — 100 petabytes (source)