– How Amazon Followed Google Into the World of Secret Servers (Wired, Nov 30, 2012):
Chris Pinkham was walking through a data center that would one day house Amazon’s seminal cloud computing service — the Elastic Compute Cloud — when he came face to face with a cage of Google machines.
This was a decade ago, when Pinkham oversaw the hardware and software that ran Amazon, and the company was considering a spot in the data center, which housed machines for many web operations and other businesses. Google would later pull a curtain around its data-center hardware, moving much of it into private facilities, but in those days, it was easier for competitors like Pinkham to lay their eyes on Google machines.
Pinkham was struck by how different the machines looked — and how hot they were. Even then, Google was running its website on dirt-cheap, stripped-down servers slotted into extremely tight spaces. They didn’t even have plastic cases.
“They were clearly not your average Dell, HP, IBM servers. They were white box machines, very densely packed. They weren’t in containers. They were just blades jammed into these custom racks,” remembers Pinkham, who went on to lead the team that built the Elastic Compute Cloud and now runs a cloud software startup called Nimbula. “And I remember a lot of heat coming off them — an indication of a lot of concentrated power.”
It was a moment that would push Pinkham and Amazon in a new direction. Inspired by Google, Pinkham says, the company began to rethink the way it used servers, moving away from the big, beefy, expensive machines, toward cheaper gear that would let the company expand its operation far more quickly. “Google was a very impressive company to us — and a bit of a role model for how to run infrastructure at scale,” he says. “I and others spent a lot of time pondering on what those servers meant.”
At first, he says, Amazon moved to the cheapest, simplest gear available from its primary server vendor, HP. But after Pinkham left the company, according to others familiar with the company’s practices, it embraced smaller vendors who could help cut out a bit more of the fat, such as Rackable Systems and ZT Systems.
And now, according to James Hamilton, the man who oversees Amazon’s current data centers, the company is building its own servers in tandem with Asian manufacturers along the lines of Quanta and Foxconn, the outfit that famously builds Apple iPhones and iPads. Hamilton tells Wired that Amazon buys its server processors and memory directly from Intel, doing an end-run around middle men such as HP and Dell and other original equipment manufacturers, or OEMs.
Once you reach a certain size, says Hamilton, it only makes sense to build your own gear. Buying traditional servers gets too expensive. If you’re buying enough hardware to negotiate favorable deals with the likes of Intel — and you’ve got the means to hire people who can run this kind hardware operation — you can significantly cut costs by going straight to Asia.
The result is an enormous shadow market for servers and other data-center hardware that’s hidden from those who traditionally track hardware sales. Google goes straight to Asia. So does Facebook. And according to a former Quanta employee who spoke to Wired earlier this year, even Microsoft is purchasing data-center hardware straight from Asian companies that the average American has never heard of.
What’s more, these practices are beginning to change how many other companies are operating — including companies beyond the web game. the financial giant Goldman Sachs recently told Wired that it buys gear directly from Asian manufacturers such as Quanta — the same manufacturers that build gear on behalf of American server sellers such as Dell and HP. And with Facebook open sourcing the designs of its custom servers, the trend is trickling down to smaller operations looking to save cost — and power — inside the data center.
Jeff Hammerbacher — who founded the data team at Facebook and co-founded the data-software startup Cloudera — is building a server cluster to run data analysis for New York’s Mount Sinai hospital, and he’s considering Facebook’s designs. As he explores this option, he’s also evaluating gear from ZT Systems — another Amazon source — in an effort to save cost.
In short, the hardware landscape is changing in a big way. Earlier this year, Diane Bryant, the head of Intel’s data center group, told us that eight server makers now account for 75 percent of Intel’s server chip revenues — and one of those is Google. Just four years ago, three companies made up that 75 percent: Dell, HP, and IBM.
According to a source familiar with Facebook’s operation, the social-networking giant also buys its chips straight from Intel, joining Google and Amazon in this camp. Google’s operation is significantly larger than Amazon’s or Facebook’s, but with the rise of the Elastic Compute Cloud, or EC2 — a service that offers up virtual servers where anyone can run just about any software they want — Amazon may not be far behind.
The trend also extends beyond servers. Google, Amazon, and Microsoft are purchasing networking switches directly from Asian manufacturers such as Quanta, as a former Quanta employee told us earlier this year.
In operating at such an enormous scale and cutting hardware costs in this way, Amazon can improve its bottom line, but it can also drive down the cost of the internet services it offers to the rest of the world, such as EC2 and other Amazon Web Services. As University of California, Berkeley, computer science professor David Patterson points out, Amazon Web Services were remarkably cheap from the very beginning, and the prices continue to drop. This week, at its first AWS conference in Las Vegas, Amazon announced a 25 percent price cut on S3, its online storage service.
Amazon is running its web operation in much the same way it runs its famous retail business. Ultimately, EC2 is just selling a commodity. So many others can sell the same thing. In order to make it work, you have to operate on very low financial margins. “Amazon, from our retail upbringings, has this background and this comfort and this skill in running high-volume, low-margin businesses,” says Andy Jassy, who oversees EC2 and its sister Amazon Web Services. “We have that DNA.”
Edward Lazowska — a computer science and engineering professor at the University of Washington who closely follows Amazon and its web services — agrees. “These are guys that are unbelievably good at operating on razor-thin margins,” he says. “Everything is relentlessly optimized for efficiency.”
Of course, when it comes to streamlining its hardware, Amazon is following in the footsteps of Google — and Google has followed Amazon with its own cloud computing service: Google Compute Engine. With their internet services, Google and Amazon are giving the world an even easier — and cheaper — way of running their operations without the likes of HP and Dell. They provide virtual servers you can use from any web browser.
“It will be interesting to see, over the next 10 years or so, how successful the traditional server vendors will be competing against that kind of server capacity,” Pinkham says. “Once developers realize they can use this much cheaper, homogenous infrastructure, the power may shift toward the folks who build the cheapest, simplest hardware.”
In other words, as the world moves to virtual servers run by the likes of Google and Amazon, it can’t help but move to the low-cost, straight-from-Asia machines that underpin those virtual servers.
Cloud computing changes things in more ways than one.