NSA’s Utah Spy Supercenter Crippled By Power Surges

– NSA’s Utah Spy Supercenter Crippled By Power Surges (ZeroHedge, Oct 8, 2013):

Long before Edward Snowden’s whistleblowing revelations hit the world and the Obama administration’s approval ratings like a ton of bricks, we ran a story in March 2012 which exposed the NSA’s unprecedented domestic espionage project, codenamed Stellar Wind, and specifically the $1.4+ billion data center spy facility located in Bluffdale, Utah, which spans more than one million square feet, uses 65 megawatts of energy (enough to power a city of more than  20,000), and can store exabytes or even zettabytes of data (a zettabyte is 100 million times larger than all the printed material in the Library of Congress), consisting of every single electronic communication in the world, whether captured with a warrant or not. Yet despite all signs to the contrary, Uber-general Keith Alexander and his spy army are only human, and as the WSJ reports, the NSA’s Bluffdale data center – whose interior may not be modeled for the bridge of the Starship Enterprise – has been hobbled by chronic electrical surges as a result of at least 10 electrical meltdowns in the past 13 months.

The facility above is where everyone’s back up phone records and emails are stored.

Such meltdowns have prevented the NSA from using computers at its new Utah data-storage center which then supposedly means that not every single US conversation using electronic media or airwaves in the past year has been saved for posterity and the amusement of the NSA’s superspooks.

This being the NSA, of course, not even a blown fuse is quite the same as it would be in the normal world: “One project official described the electrical troubles—so-called arc fault failures—as “a flash of lightning inside a 2-foot box.” These failures create fiery explosions, melt metal and cause circuits to fail, the official said. The causes remain under investigation, and there is disagreement whether proposed fixes will work, according to officials and project documents. One Utah project official said the NSA planned this week to turn on some of its computers there.”

More from the WSJ on this latest example of what even the most organized and efficient of government agencies ends up with when left to its non-private sector resources:

Without a reliable electrical system to run computers and keep them cool, the NSA’s global surveillance data systems can’t function. The NSA chose Bluffdale, Utah, to house the data center largely because of the abundance of cheap electricity. It continuously uses 65 megawatts, which could power a small city of at least 20,000, at a cost of more than $1 million a month, according to project officials and documents.

Utah is the largest of several new NSA data centers, including a nearly $900 million facility at its Fort Meade, Md., headquarters and a smaller one in San Antonio. The first of four data facilities at the Utah center was originally scheduled to open in October 2012, according to project documents. The data-center delays show that the NSA’s ability to use its powerful capabilities is undercut by logistical headaches. Documents and interviews paint a picture of a project that cut corners to speed building.

Backup generators have failed numerous tests, according to project documents, and officials disagree about whether the cause is understood. There are also disagreements among government officials and contractors over the adequacy of the electrical control systems, a project official said, and the cooling systems also remain untested.

The Army Corps of Engineers is overseeing the data center’s construction. Chief of Construction Operations, Norbert Suter said, “the cause of the electrical issues was identified by the team, and is currently being corrected by the contractor.” He said the Corps would ensure the center is “completely reliable” before handing it over to the NSA.

But another government assessment concluded the contractor’s proposed solutions fall short and the causes of eight of the failures haven’t been conclusively determined. “We did not find any indication that the proposed equipment modification measures will be effective in preventing future incidents,” said a report last week by special investigators from the Army Corps of Engineers known as a Tiger Team.

The architectural firm KlingStubbins designed the electrical system. The firm is a subcontractor to a joint venture of three companies: Balfour Beatty Construction, DPR Construction and Big-D Construction Corp. A KlingStubbins official referred questions to the Army Corps of Engineers.

Tsk tsk: this is what happens when you use taxpayer dollars to pay the lowest bidder – you can’t even build an efficient totalitarian superstate!

The first arc fault failure at the Utah plant was on Aug. 9, 2012, according to project documents. Since then, the center has had nine more failures, most recently on Sept. 25. Each incident caused as much as $100,000 in damage, according to a project official. It took six months for investigators to determine the causes of two of the failures. In the months that followed, the contractors employed more than 30 independent experts that conducted 160 tests over 50,000 man-hours, according to project documents.

This summer, the Army Corps of Engineers dispatched its Tiger Team, officials said. In an initial report, the team said the cause of the failures remained unknown in all but two instances. The team said the government has incomplete information about the design of the electrical system that could pose new problems if settings need to change on circuit breakers. The report concluded that efforts to “fast track” the Utah project bypassed regular quality controls in design and construction.

Contractors have started installing devices that insulate the power system from a failure and would reduce damage to the electrical machinery. But the fix wouldn’t prevent the failures, according to project documents and current and former officials.

Contractor representatives wrote last month to NSA officials to acknowledge the failures and describe their plan to ensure there is reliable electricity for computers. The representatives said they didn’t know the true source of the failures but proposed remedies they believed would work. With those measures and others in place, they said, they had “high confidence that the electrical systems will perform as required by the contract.”

A couple of weeks later, on Sept. 23, the contractors reported they had uncovered the “root cause” of the electrical failures, citing a “consensus” among 30 investigators, which didn’t include government officials. Their proposed solution was the same device they had already begun installing.

Wait, we know: it’s the Syrians.

So for those who have no choice but to live in a totalitarian banana republic, may we suggest at least laughing about it. We present the Domestic Surveillance Directorate.