Will NSA's Utah Data Center be able to handle and process five zettabytes of data?

Question

The claim I want to have examined is

The NSA's Utah Data Center will be able to handle and process five zettabytes of data.

I think that "zettabytes" is much too big. Even one Exabyte might be to much to store (not even to speak about the possibility to process such an amount of data).

Sources

The Utah Data Center, also known as the Intelligence Community Comprehensive National Cybersecurity Initiative Data Center, is a data storage facility for the United States Intelligence Community that is designed to store extremely large amounts of data.

Source: Wikipedia

The National Security Agency is building a data centre that could potentially hold yottabytes of data.

Source: cnet.co.uk

The estimated power of those computing resources in Utah is so massive it requires use of a little-known unit of storage space: the zettabyte. Cisco quantifies a zettabyte as the amount of data that would fill 250 billion DVDs. [...] The NSA's Utah Data Center will be able to handle and process five zettabytes of data, according to William Binney, a former NSA technical director turned whistleblower. Binney's calculation is an estimate. An NSA spokeswoman says the actual data capacity of the center is classified.

Source: npr.org

Space(1 DVD) = 4.7 GB
Space(250 * 10^9 DVDs) = 250*10^9*4.7 GB = 1175 EB = 1.175 ZB

As those numbers roughly match, I guess it's not a simple typo on npr.org.

But 1 YB = 1000 ZB, so this doesn't match.

Also: The biggest storage device I could find on Amazon with this search was Quantum StorNext AEL500 Archive for $143,918.20. It can store 399 TB. This means 399 TB / $143,918.20 = $0.36 per GigaByte

So for 5 ZB we would need (5 * 10^21 Byte) / (399 * 10^12 Byte) = 1.253*10^7. That would mean you would have to pay about $143,918.20*1.253*10^7 = $1.803 trillion (US dollars). But:

The U.S. Army Corps of Engineers broke ground this week on a massive new National Security Agency cyber intelligence center in Utah. Located at Camp Williams, 25 miles south of Salt Lake City, the $1.2 billion facility — officially known as the Utah Data Center — will be responsible for collecting and aggregating incoming intelligence data.

Source: defensesystems.com

Estimates

Even after searching for a device with lowest dollar-per-gigabyte ratio, I didn't find anything cheaper than $0.05/GB. This means you could come down to 250 billion US-$, which is still much too much.

Even more problematic might be the power consumption.

Questions that might lead to an answer

Did I make any mistake? Could this eventually be true (Whats hot in science about data storages?) Which capacity do big companies like Google / Facebook / Dropbox have? How much can Wikipedia (especially commons) store?

Storage systems

Provide up to 900 PB of automated, low-cost storage

IBM System Storage TS3500 Tape Library

Problems of this question

• I thought "handle and process" would mean they have to store this amount of information, but the source doesn't say that
• It's not clear what "handle and process" exactly means. Especially throughput should be mentioned.

Answer 1

This answer is highly speculative since the NSA has not given any indication as to the capacity of the facility. Also, this is a very difficult question to digest because we have no indication of how much data they are seeking to actually store.

To examine the first claim that they process up to five zettabytes of data, this seems to be plausible. We know that the Large Hadron Collider (LHC) is generating one petabyte data per second and storing twenty-five petabytes of data per year¹. Using some simple math,

1 zettabyte = 1,048,576 petabytes
1,048,576 seconds = 12.1363 days

1 zettabyte every 12.1363 days 
or about 30 zettabytes of data a year

According to Cisco's forecasting, the world is on track to have 1.4 zettabytes of global IP traffic per year by the end of 2017. So anyone that is engaged in processing that data data would need to have zettabyte processing capability and given that the LHC already has it, we can conclude that it is likely that an agency such as the NSA would design that into the system. Also, that just includes the IP traffic and other voice and data communications that travel via other systems.

With regards to storage of data, the common number seems to be five zettabytes so we will examine that. The capability for exabyte storage system already exists commercially. IBM manufactures the System Storage TS3500 Tape Library offers up to 2.7 exabytes of data, specifically,

Up to 180 PB compressed with 3592 extended capacity cartridges per library, up to 2.7 EB compressed per complex

The physical footprint of a single frame is 70.9 in. H × 30.8 in. W × 47.7 in. D (1,800 mm × 782 mm × 1,212 mm) or a footprint of about 10.2 sq.ft. (0.9 sq.m.) which puts it in range of most 42U rack dimensions. In order to reach the full 2.7 exabytes, 15 libraries are needed, each of which would need 16 frames which gives us 2,448 sq.ft (227.4 sq.m.). The amount of additional space needed due to airflow and system access is fairly complex, so excluding that from the equation,

2.7 exabytes per complex with a footprint of 2,448 sq.ft.
1 zettabyte = 1024 exabytes
1024 / 2.7 = 379.2 systems, so say 380 systems
380 * 2,448 sq.ft = 930,240 sq.ft per zettabyte

This gives us about 4,651,200 sq.ft. (432,110.62 sq.m) for the 5 zettabytes figure normally given. Even storage of a single zettabyte using the systems appears to be outside of the bounds of 100,000 sq.ft. for the data halls that the Utah Data Center will have let alone the 1 to 1.5 million sq.ft. size given for the complex as a whole.

Based upon this, the claims that it will be storing five zettabytes of data is dubious; however, these calculations are based upon a commercially available system so it is possible that they have a smaller footprint for what they are using. That said though, even storage of a single zettabyte is likely an extreme stretch given what is on the commercial market. However, in 2010 IBM Research announced a new record in tape storage with a potential capacity of 35 terabytes of uncompressed storage per cartridge. The current high-end offering from IBM is the 3592 Tape Cartridge which stores up to 4 terabytes.

The aforementioned TS3500 Tape Library holds up to 225,000 tape cartridges so if each of the cartridges was capable of 35 terabytes the total TS3500 complex storage would be about 7.51 exabytes of storage. If we apply these updated values to the previous calculations,

7.51 exabytes per complex with a footprint of 2,448 sq.ft.
1 zettabyte = 1024 exabytes
1024 / 7.51 = 136.4 systems, so say 137 systems
137 * 2,448 sq.ft = 335,376 sq.ft per zettabyte

So even with that the data halls are still too small to store a single zettabyte although we are now into the range where the complex as a whole might be capable of storing one to two zettabytes under ideal conditions.

Granted does require the use of tape data storage which is very slow; however, without knowing exactly what the NSA is doing with the data we can't speculate if they are concerned with access speeds or not. Furthermore, metadata indexing techniques mean that the raw data may not be needed for most processing so the latency involved with tape storage may not be a concern for the NSA. We already know that they have large scale metadata databases so it is quite likely that they could be using such techniques for working with tape data.

So in summary, the claims as to the amount of data that could be stored and processed are currently implausible given the size of the facility and the currently know state of the art and commercially available systems. However, the claim that the could be processing five zettabytes of data is plausible given the shear amount of data traffic that exists in the world and we don't know how they are processing the data. If they use curiosity examination followed by discarding of uninteresting data like the LHC then it is possible and their actual data storage requirements would be reduced as well.

---

1. In the case of the LHC, they discard most of the data as "uninteresting" as opposed to actually storing it.
2. Even if the data halls are only 100,000 sq.ft. there still should be sufficient space although it might be less likely given that we don't know the configuration of the buildings.

Answer 2

There are several claims about extremely huge data storage capacity in Utah, like "The NSA's Utah Data Center will be able to handle and process five zettabytes of data." (yotta = 1000 zetta = 1M exa = 1 billion peta = 1 trillion tera)

Many of them comes from wrong estimations from William Binney (a former NSA technical director), according to NPR: http://www.npr.org/2013/06/10/190160772/amid-data-controversy-nsa-builds-its-biggest-data-farm

The NSA's Utah Data Center will be able to handle and process five zettabytes of data, according to William Binney, a former NSA technical director turned whistleblower. Binney's calculation is an estimate. An NSA spokeswoman says the actual data capacity of the center is classified.

Here is full photocopy of Binney declaration for the CASE NO. CV-08-04373-JSW, from September 28, 2012: https://www.eff.org/sites/default/files/filenode/binneydeclaration.pdf - page 4 lines 18-26:

18   11. A futher notable development has been the NSA's public announcement in October 
19  2009 that it was building a massive, $1.2 billion digital storage facility in Ft. Williams, Utah.
20  According to some reports, the Utah facility will eventually have a data storage capacity measured 
21  in yottabytes (10**24 bytes). Even if the Utah facility were to have no more than the amount of data
22  storage that is presently commercially available, then one would expect the data storage to be in the
23  range of multiples of ten exebytes (10**18 bytes). See www.cleversafe.com. (According to 
24  Cleversafe, its ten exebyte storage solution fills no more than tho hundred square feet).

Forbes checked his estimations and found huge error in how much space will Cleversafe's solution needs for 10 exabytes: http://www.forbes.com/sites/kashmirhill/2013/07/24/blueprints-of-nsa-data-center-in-utah-suggest-its-storage-capacity-is-less-impressive-than-thought/

Blueprints Of NSA's Ridiculously Expensive Data Center In Utah Suggest It Holds Less Info Than Thought, 7/24/2013

Binney gave the 5 zettabyte estimate to NPR and also included it in an affidavit filed in Jewel vs. NSA ...

His estimate is based on the assumption the facility might offer equipment like that developed by Cleversafe. The company says it has a 10-exabyte data storage system that involves portable data centers with 21 racks each

Binney read this as meaning that 21 racks could hold 10 exabytes, and assumed an efficient rack size of 4 square feet, ...

he misread the Cleversafe marketing materials. They actually say that 560 portable data centers of 21 racks each (or 11,760 racks) can hold 10 exabytes. ...

Chris Gladwin, founder of Cleversafe, says that in January of 2012, 10 exabytes of storage system would have needed “about 2 million square feet.”

So, marketing speech of Cleversafe leads ex NSA technical director to the very wrong estimations.

Answer 3

Depends on the interpretation of what handle and process five zettabytes of data really means:

1. having 5ZB readily available for online processing?
2. having 5ZB in off-line archive, while having only meta-data readily available for on-line processing?
3. having 5ZB incoming traffic, discharging most of it, storing and processing only interesting stuff?

On-line storage

If you're going to build something as big as NSA datacenter, instead of buying readily available hardware, you might go for custom builds using big data industry standards.

Facebook's Open Compute Storage specifies Open Vault storage, which is designed to house 30 3.5" HDDs in 2U case. Which with currently available HDDs would mean density of 120TB per 2U.

You can put 21 of these in one standard 42U high 19" rack, meaning you'd have 2520TB per rack.

Each rack having a base of about 0.62m². Adding some very minimum clearance, 1m² per rack would be the very lowball estimate.

That's assuming that you'd be willing to have machines custom build for your datacenter. Otherwise it's off the shelf hardware. For example Dell PowerVault MD1200 configured with 12 4TB SATA drives costs $11,254.21; That of course is retail price, which bulk orders you get hefty discounts. Similarly, you could get a cloud server such as PowerEdge C6145, which can also be equipped with 12 4TB SATA drives. They don't provide they pricing on-line, but if I remember correctly, that kind of setup it'd be around $15-20K.

Both of above give you density of 48TB per 2U. Which means 1004TB per rack.

NSA Utah datacenter is reported to have 10,000m² dedicated to servers (100,000m² including administrative and support buildings). So really, really high ball estimate would be that they could theoretically squeeze 25EB of cloud storage in there. That's only 0.5% of 5ZB.

Off-line storage

If you're considering offline archiving of data, you might use tape libraries. It's not typically used in Big Data, but then again, NSA isn't typical Big Data company.

Problem is, that even with highest density IBM System Storage TS3500 Tape Library, you can store 180PB per library, which has footprint of at least 15m². Which means about 420,000m² would be needed just for the tape libraries. Whole NSA Utah complex is estimated to have "1 million or 1.5 million square feet" (93,000m²-149,000m²), but it's also been said that it's "100,000 square feet (9,300m²) of data center space and greater than 900,000 square feet (83,600m²) of technical support and administrative space" (source). Purpose of the buildings might be kept secret or intentionally misleading, external dimensions cannot. So clearly there is no way to squeeze 5ZB of tape archive there, even if buildings said to be administrative are actually purposed for tape archive.

Meta-data

So let's assume that even if you'd manage to somehow archive 5ZB of raw data, as I've calculated above even if you'd use all your datacenter space for on-line processing, that would mean you're storing only 5KB of metadata for each 1MB of data. While for media files that seems more than enough, I really doubt you can achieve ratio anywhere near that for emails, chats, text messages, tweets etc. Of course they might filter out information they've not interested in and not store any metadata related to that information. But in that case there is no point in archiving that information off-line, because you don't have path of access — you don't have metadata pointing to it, nor can you mine the data that is archived on tapes.

Throughput

Assuming that it's 5ZB per year, that's a sustained throughput of 570PB per day, or 160TB per second. Leaving apart computing power needed to process that, question is how that data would be transported there? Current internet infrastructure isn't anywhere nearly ready for that kind of traffic. Three of the biggest internet exchange points are located in Europe having maximum throughput of 2.2TBps, 2.1TBps and 1.6TBps respectively. US is trailing way behind with top US only exchanges having only 0.28TBps and 0.25TBps, another one, Equinix has total of 1.4TBps across 12 countries on 4 continents. In other words it's completely infeasible to push 5ZB per year into a datacenter, especially in such a remote location. Nor, as you can see, necessary, even if you'd want to process all of Internet traffic.

Conclusion

1. having 5ZB readily available for online processing?
   Physically impossible
2. having 5ZB in off-line archive, while having only meta-data readily available for on-line processing? Not possible with current technology and sizes of the buildings as reported
3. having 5ZB incoming traffic, discharging most of it, storing and processing only interesting stuff? Regardless of processing, it's just not possible to have that amount of incoming traffic

---

HighScalability.com has a guest post "PRISM: The Amazingly Low Cost Of ­Using BigData To Know More About You In Under A Minute" by BugSense Founder/CTO Jon Vlachogiannis and Head of Infrastructure at BugSense Panagiotis Papadomitsos.

They calculate for storage needed as 3.75EB.

DATA

Facebook: 500 TB/day * 30 = 1.5 PT/month (source)

Twitter: 8 TB/day * 30 = 240 TB/month 8 TB/day (source)

Email/Other info: 193PT/month Google says 24 PB per day (2008). Five years later lets assume this is 8 times bigger = 192 PB. Now, real user information is 1/3 = 64 PT/day (source)

Mobile traffic/machine­to­machine exchanges/vehicles etc: 4000 TB per day = 117 PB/month (source)

Total Data =~312PB month

Hardware Costs

The prices below correspond to renting off­the­shelf servers from commercial high­end datacenters (considering the data will be stored in a distributed filesystem architecture such as HDFS). This is a worst case scenario that does not include potential discounts due to renting such a high volume of hardware and traffic or acquiring the aforementioned hardware (which incurs a higher initial investment but lower recurring costs) . The hardware configuration used for calculating costs in this case study is comprised of a 2U chassis, dual Intel Hexa­core processors, 16 GB of RAM, 30 TB of usable space combined with hardware­level redundancy (RAID5).

We’ll be needing about 20K servers, put into 320 46U racks. Cost for the server hardware is calculated to be about €7.5M / month (including servers for auxiliary services). Cost for the racks, electricity and traffic is calculated to be about €0.5M / month (including auxiliary devices and networking equipment).

Total hardware cost per year for 3.75 EB of data storage: €168M

---

And comparison with Facebook:

It’s no secret that Facebook stores a lot of data — 100 petabytes (source)

Answer 4

Based on rough calculations done by Brewster Kahle who knows about storage from the Internet Archive, about 270 Petabytes (and $27M) is needed to store all US phone calls for a year. So, these numbers may be more in the order of magnitude for expectations of the Utah data center in the near term. http://blog.archive.org/2013/06/15/cost-to-store-all-us-phonecalls-made-in-a-year-in-cloud-storage-so-it-could-be-datamined/