Long time back (Couple of years..) one of my colleague mentioned to me that every search on google generates enough heat that you can boil water for your cup of a tea.. I thought that he was joking or at least exaggerating.. but reading all this news.. doesn't surprise me that it was right information.
Worldwide.. Data Centers now consume around 30,000 MW at any point of time... Just for comparison, India's total power generation capacity must be around 200,000 MW.. California's total power generation capacity is around 70,000 MW (give and take 5-10K MW here and there..)..
Google alone consumes 300MW for their Data Centers(DC)/Server farms.. Poor Facebook.. just 60MW.
One MW is Mega Watt and equals to One Million Watts. Your iPhone charger consumes 5W when charging your iPhone. MW or Mega Watt is common terminology for utility industry to calculate generation capacity or consumption capacity.. either way they are more or less same as you really can't store electricity at this level. Now a days we have started using GW (Giga Watts) which is equal to 1000 MW or one Billion Watts. that means.. that you can charge 200 Million iPhones in 1000MW :-)
Okay.. back to main topic.. Agreed that Facebook, Gmail or Google search are as necessary as "Roti", Kapda" & "Makaan" (Food, cloth and house). and this is the reason.. all these big data center monsters are not only consuming 30K MW all the time but also wasting almost 90% of their capacity in anticipation of surge in demand or in anticipation that grid will fail and there will be power outage which will make their site down.. Mostly for user's like you and mine's satisfaction!!! or in other words to protect their reputation!! Nobody wants news that their web site was down for this many hours or slow today.. etc..
First of all, I think, this claim of 90% wastage.. I think it is way to exaggerated.. I agree.. There is lot of wastage but I don't think that it will be close to 90%.. Still.. there is no smoke without fire.. so let us assume that there is 50-60% wastage.. Which is still huge.. 15,000 MW!!!
Then secondly, can we do something to cut down this wastage. As per these Data Center / Server Farm Monsters.. all this wastage is for me.. and bad news is that these guys have made me addicted to instant web/information so much that I can't live without it.. I can try to cut it down or go on web diet.. That will make some dent or not.. not known.. but definitely.. if every one decides.. then it will.. I will be boiling lesser number of tea pots.. Honestly, I don't think that it will be possible to cut my Google/Facebook usage... they have become more or less my lifelines.. (that was too much..). but I can try..
Lastly, major reason for all this high level of consumption and this alleged wastage is that service level expectations for me as an user are really high.. do I care if Google returns search results in some 0.0003 seconds (just kidding) or whatever number they show.. or do I care if FB page loads slightly slower or shows my friend's updates in 10 minutes instead of 0.0000000004 seconds (again just kidding..).. What will happen if Google or FB or other such monsters give me option of being "Green User" I get classified for most of my work in Green category which could have SLA issues here and there and instead of returning search results in 0.0003 seconds can return results in 0.3 or even 3 seconds... or at least wait for me to completely type my search request instead of searching for each and every letter I type and thus boiling a whole barrel of water instead of a tea pot..
I won't mind these SLAs.. I waste enough time all around here and there and I could be fine.. If I am under pressure to perform.. rarely but possible.. then I should be able to change my profile from Green to Red Hot user who would like super fast response like I get today..
They can give me some kind of special green badge (the kind of green ranger badge kids get in yosemite national park for attending some environmental session by rangers ;-) ) in lieu and I can proudly use it in my email signature to let the world know about it.. so that the world would stop expecting immediate email response or immediate FB like or comment on their post!!!
Let me know what all you guys think about it.. I was thinking about this article for couple of days and wanted to write it so it comes out effective. I tried my best to keep it simple and humorous where possible..
depending upon the response from web (It is okay if we boil few more barrels of water.. we will save some tankers later on..), I was thinking that we can start some Google+ or FB+ page for this cause and start some awareness.. so we can get some kind of petition for Google and FB and other web monsters to start some kind of "Green User" concept who will not complain of average or below average SLAs for their services as long as this user is not part of this wasteful Data Center (DC) or Server Farm.
I will be happy on some kind of green and energy efficient server farm or data center, where electricity is consumed to serve but wastage is significant lower.. I will take little bit of performance hit on my side.. I am sure.. with technological advancements.. this will be a short lived process.. Very soon DC technologies will improve significantly where they can add or reduce computing capacity on demand without significant hit on performance SLA.
But for the time being.. I am ready to take this hit.. are you joining me in this?
The number of data centers is surging.
Google alone uses 300 million watts.
RICHARD PERRY/ NEW YORK TIMES ARCHIVES
=========================================================================
Worldwide.. Data Centers now consume around 30,000 MW at any point of time... Just for comparison, India's total power generation capacity must be around 200,000 MW.. California's total power generation capacity is around 70,000 MW (give and take 5-10K MW here and there..)..
Google alone consumes 300MW for their Data Centers(DC)/Server farms.. Poor Facebook.. just 60MW.
One MW is Mega Watt and equals to One Million Watts. Your iPhone charger consumes 5W when charging your iPhone. MW or Mega Watt is common terminology for utility industry to calculate generation capacity or consumption capacity.. either way they are more or less same as you really can't store electricity at this level. Now a days we have started using GW (Giga Watts) which is equal to 1000 MW or one Billion Watts. that means.. that you can charge 200 Million iPhones in 1000MW :-)
Okay.. back to main topic.. Agreed that Facebook, Gmail or Google search are as necessary as "Roti", Kapda" & "Makaan" (Food, cloth and house). and this is the reason.. all these big data center monsters are not only consuming 30K MW all the time but also wasting almost 90% of their capacity in anticipation of surge in demand or in anticipation that grid will fail and there will be power outage which will make their site down.. Mostly for user's like you and mine's satisfaction!!! or in other words to protect their reputation!! Nobody wants news that their web site was down for this many hours or slow today.. etc..
First of all, I think, this claim of 90% wastage.. I think it is way to exaggerated.. I agree.. There is lot of wastage but I don't think that it will be close to 90%.. Still.. there is no smoke without fire.. so let us assume that there is 50-60% wastage.. Which is still huge.. 15,000 MW!!!
Then secondly, can we do something to cut down this wastage. As per these Data Center / Server Farm Monsters.. all this wastage is for me.. and bad news is that these guys have made me addicted to instant web/information so much that I can't live without it.. I can try to cut it down or go on web diet.. That will make some dent or not.. not known.. but definitely.. if every one decides.. then it will.. I will be boiling lesser number of tea pots.. Honestly, I don't think that it will be possible to cut my Google/Facebook usage... they have become more or less my lifelines.. (that was too much..). but I can try..
Lastly, major reason for all this high level of consumption and this alleged wastage is that service level expectations for me as an user are really high.. do I care if Google returns search results in some 0.0003 seconds (just kidding) or whatever number they show.. or do I care if FB page loads slightly slower or shows my friend's updates in 10 minutes instead of 0.0000000004 seconds (again just kidding..).. What will happen if Google or FB or other such monsters give me option of being "Green User" I get classified for most of my work in Green category which could have SLA issues here and there and instead of returning search results in 0.0003 seconds can return results in 0.3 or even 3 seconds... or at least wait for me to completely type my search request instead of searching for each and every letter I type and thus boiling a whole barrel of water instead of a tea pot..
I won't mind these SLAs.. I waste enough time all around here and there and I could be fine.. If I am under pressure to perform.. rarely but possible.. then I should be able to change my profile from Green to Red Hot user who would like super fast response like I get today..
They can give me some kind of special green badge (the kind of green ranger badge kids get in yosemite national park for attending some environmental session by rangers ;-) ) in lieu and I can proudly use it in my email signature to let the world know about it.. so that the world would stop expecting immediate email response or immediate FB like or comment on their post!!!
Let me know what all you guys think about it.. I was thinking about this article for couple of days and wanted to write it so it comes out effective. I tried my best to keep it simple and humorous where possible..
depending upon the response from web (It is okay if we boil few more barrels of water.. we will save some tankers later on..), I was thinking that we can start some Google+ or FB+ page for this cause and start some awareness.. so we can get some kind of petition for Google and FB and other web monsters to start some kind of "Green User" concept who will not complain of average or below average SLAs for their services as long as this user is not part of this wasteful Data Center (DC) or Server Farm.
I will be happy on some kind of green and energy efficient server farm or data center, where electricity is consumed to serve but wastage is significant lower.. I will take little bit of performance hit on my side.. I am sure.. with technological advancements.. this will be a short lived process.. Very soon DC technologies will improve significantly where they can add or reduce computing capacity on demand without significant hit on performance SLA.
But for the time being.. I am ready to take this hit.. are you joining me in this?
DATA CENTERS’ DIRTY SECRET
Wasting electricity 24/7
Server farms run full-bore even when demand is low
By James Glanz
New York Times
SANTA CLARA — Jeff Rothschild’s machines at Facebook had a problem he knew he had to solve immediately. They were about to melt.
The company had been packing a 40-by-60-foot rental space here with racks of computer servers that were needed to store and process information from members’ accounts. The electricity pouring into the computers was overheating Ethernet sockets and other crucial components.
Thinking fast, Rothschild, the company’s engineering chief, took some employees on an expedition to buy every fan they could find — “We cleaned out all of the Walgreens in the area,” he said — to blast cool air at the equipment and prevent the website from going down.
Wasting electricity 24/7
Server farms run full-bore even when demand is low
By James Glanz
New York Times
SANTA CLARA — Jeff Rothschild’s machines at Facebook had a problem he knew he had to solve immediately. They were about to melt.
The company had been packing a 40-by-60-foot rental space here with racks of computer servers that were needed to store and process information from members’ accounts. The electricity pouring into the computers was overheating Ethernet sockets and other crucial components.
Thinking fast, Rothschild, the company’s engineering chief, took some employees on an expedition to buy every fan they could find — “We cleaned out all of the Walgreens in the area,” he said — to blast cool air at the equipment and prevent the website from going down.
The number of data centers is surging.
Google alone uses 300 million watts.
RICHARD PERRY/ NEW YORK TIMES ARCHIVES
=========================================================================
That was in early 2006, when Facebook had a quaint 10 million or so users and just the one main server site. Today, the information generated by nearly 1 billion users requires outsize versions of these facilities, called data centers, with rows and rows of servers spread over hundreds of thousands of square feet, and all with industrial cooling systems.
They are a mere fraction of the tens of thousands of data centers that now exist to support the overall explosion of digital information. Stupendous amounts of data are set in motion each day as, with an innocuous click or tap, people download movies on iTunes, check credit card balances on Visa’s website, send Yahoo email with files attached, buy products on Amazon, post on Twitter or read newspapers online. A yearlong examination by The New York Times has revealed that this foundation of the information industry is sharply at odds with its image of sleek efficiency and environmental friendliness.
Most data centers, by design, consume vast amounts of energy in an incongruously wasteful manner, interviews and documents show. Online companies typically run their facilities at maximum capacity around the clock, whatever the demand. As a result, data centers can waste 90 percent or more of the electricity they pull off the grid, the Times found.
To guard against a power failure, they further rely on banks of generators that emit diesel exhaust. The pollution from data centers has increasingly been cited by the authorities for violating clean air regulations, documents show. In Silicon Valley, many data centers appear on the state government’s Toxic Air Contaminant Inventory, a roster of the area’s top stationary diesel polluters.
Worldwide, the digital warehouses use about 30 billion watts of electricity, roughly equivalent to the output of 30 nuclear power plants, according to estimates industry experts compiled for the Times. Data centers in the United States account for onequarter to one-third of that load, the estimates show.
“It’s staggering for most people, even people in the industry, to understand the numbers, the sheer size of these systems,” said Peter Gross, who helped design hundreds of data centers. “A single data center can take more power than a medium-size town.”
Energy efficiency varies widely from company to company. But at the request of the Times, the consulting firm McKinsey & Co. analyzed energy use by data centers and found that, on average, they were using only 6 to 12 percent of the electricity powering their servers to perform computations. The rest was essentially used to keep servers idling and ready in case of a surge in activity that could slow or crash their operations.
A server is a sort of bulked-up desktop computer, minus a screen and keyboard, that contains chips to process data. The study sampled some 20,000 servers in about 70 large data centers spanning the commercial gamut: drug companies, military contractors, banks, media companies and government agencies.
“This is an industry dirty secret, and no one wants to be the first to say mea culpa,” said a senior industry executive who asked not to be identified to protect his company’s reputation. “If we were a manufacturing industry, we’d be out of business straightaway.”
These physical realities of data are far from the mythology of the Internet: where lives are lived in the “virtual” world and all manner of memory is stored in “the cloud.”
The inefficient use of power is largely driven by a symbiotic relationship between users who demand an instantaneous response to the click of a mouse and companies that put their business at risk if they fail to meet that expectation.
Even running electricity at full throttle has not been enough to satisfy the industry. In addition to generators, most large data centers contain banks of huge, spinning flywheels or thousands of lead-acid batteries — many of them similar to automobile batteries — to power the computers in case of a grid failure as brief as a few hundredths of a second, an interruption that could crash the servers.
“It’s a waste,” said Dennis Symanski, a senior researcher at the Electric Power Research Institute, a nonprofit industry group. “It’s too many insurance policies.” At least a dozen major data centers have been cited for violations of air quality regulations in Virginia and Illinois alone, according to state records. Amazon was cited with more than 24 violations over a three-year period in Northern Virginia, including running some of its generators without a basic environmental permit. A few companies say they are using extensively re-engineered software and cooling systems to decrease wasted power. Among them are Facebook and Google, which also have redesigned their hardware. Still, according to recent disclosures, Google’s data centers consume nearly 300 million watts and Facebook’s about 60 million watts.
Many of these solutions are readily available, but in a risk-averse industry, most companies have been reluctant to make wholesale change, according to industry experts. Improving or even assessing the field is complicated by the secretive nature of an industry that is largely built around accessing other people’s personal data.
For security reasons, companies typically do not even reveal the locations of their data centers, which are housed in anonymous buildings and vigilantly protected. Companies also guard their technology for competitive reasons, said Michael Manos, a longtime industry executive. “All of those things play into each other to foster this closed, members-only kind of group,” said Manos, now a senior vice president for technologies at AOL.
That secrecy often extends to energy use. To further complicate any assessment, no single government agency has the authority to track the industry. In fact, the federal government was unable to determine how much energy its own data centers consume, according to officials involved in a survey completed last year. The survey did discover that the number of federal data centers grew from 432 in 1998 to 2,094 in 2010.
To investigate the industry, the Times obtained thousands of pages of local, state and federal records, some through freedom of information laws, that are kept on industrial facilities that use large amounts of energy. Copies of permits for generators and information about their emissions were obtained from environmental agencies, which helped pinpoint some data center locations and details of their operations. In addition to reviewing records from electrical utilities, the Times also visited data centers across the country and conducted hundreds of interviews with current and former employees and contractors.
Some analysts warn that as the amount of data and energy use continue to rise, companies that do not alter their practices could eventually face a shake-up in an industry that has been prone to major upheavals, including the bursting of the first Internet bubble in the late 1990s. “It’s just not sustainable,” said Mark Bramfitt, a former utility executive who now consults for the power and information technology industries. “They’re going to hit a brick wall.”
STEVE DYKES/NEW YORK TIMES ARCHIVES
A row of backup generators, inside the white housings, line the back exterior of the Facebook data center in Prineville, Ore. They’ll keep servers on in a power outage.
They are a mere fraction of the tens of thousands of data centers that now exist to support the overall explosion of digital information. Stupendous amounts of data are set in motion each day as, with an innocuous click or tap, people download movies on iTunes, check credit card balances on Visa’s website, send Yahoo email with files attached, buy products on Amazon, post on Twitter or read newspapers online. A yearlong examination by The New York Times has revealed that this foundation of the information industry is sharply at odds with its image of sleek efficiency and environmental friendliness.
Most data centers, by design, consume vast amounts of energy in an incongruously wasteful manner, interviews and documents show. Online companies typically run their facilities at maximum capacity around the clock, whatever the demand. As a result, data centers can waste 90 percent or more of the electricity they pull off the grid, the Times found.
To guard against a power failure, they further rely on banks of generators that emit diesel exhaust. The pollution from data centers has increasingly been cited by the authorities for violating clean air regulations, documents show. In Silicon Valley, many data centers appear on the state government’s Toxic Air Contaminant Inventory, a roster of the area’s top stationary diesel polluters.
Worldwide, the digital warehouses use about 30 billion watts of electricity, roughly equivalent to the output of 30 nuclear power plants, according to estimates industry experts compiled for the Times. Data centers in the United States account for onequarter to one-third of that load, the estimates show.
“It’s staggering for most people, even people in the industry, to understand the numbers, the sheer size of these systems,” said Peter Gross, who helped design hundreds of data centers. “A single data center can take more power than a medium-size town.”
Energy efficiency varies widely from company to company. But at the request of the Times, the consulting firm McKinsey & Co. analyzed energy use by data centers and found that, on average, they were using only 6 to 12 percent of the electricity powering their servers to perform computations. The rest was essentially used to keep servers idling and ready in case of a surge in activity that could slow or crash their operations.
A server is a sort of bulked-up desktop computer, minus a screen and keyboard, that contains chips to process data. The study sampled some 20,000 servers in about 70 large data centers spanning the commercial gamut: drug companies, military contractors, banks, media companies and government agencies.
“This is an industry dirty secret, and no one wants to be the first to say mea culpa,” said a senior industry executive who asked not to be identified to protect his company’s reputation. “If we were a manufacturing industry, we’d be out of business straightaway.”
These physical realities of data are far from the mythology of the Internet: where lives are lived in the “virtual” world and all manner of memory is stored in “the cloud.”
The inefficient use of power is largely driven by a symbiotic relationship between users who demand an instantaneous response to the click of a mouse and companies that put their business at risk if they fail to meet that expectation.
Even running electricity at full throttle has not been enough to satisfy the industry. In addition to generators, most large data centers contain banks of huge, spinning flywheels or thousands of lead-acid batteries — many of them similar to automobile batteries — to power the computers in case of a grid failure as brief as a few hundredths of a second, an interruption that could crash the servers.
“It’s a waste,” said Dennis Symanski, a senior researcher at the Electric Power Research Institute, a nonprofit industry group. “It’s too many insurance policies.” At least a dozen major data centers have been cited for violations of air quality regulations in Virginia and Illinois alone, according to state records. Amazon was cited with more than 24 violations over a three-year period in Northern Virginia, including running some of its generators without a basic environmental permit. A few companies say they are using extensively re-engineered software and cooling systems to decrease wasted power. Among them are Facebook and Google, which also have redesigned their hardware. Still, according to recent disclosures, Google’s data centers consume nearly 300 million watts and Facebook’s about 60 million watts.
Many of these solutions are readily available, but in a risk-averse industry, most companies have been reluctant to make wholesale change, according to industry experts. Improving or even assessing the field is complicated by the secretive nature of an industry that is largely built around accessing other people’s personal data.
For security reasons, companies typically do not even reveal the locations of their data centers, which are housed in anonymous buildings and vigilantly protected. Companies also guard their technology for competitive reasons, said Michael Manos, a longtime industry executive. “All of those things play into each other to foster this closed, members-only kind of group,” said Manos, now a senior vice president for technologies at AOL.
That secrecy often extends to energy use. To further complicate any assessment, no single government agency has the authority to track the industry. In fact, the federal government was unable to determine how much energy its own data centers consume, according to officials involved in a survey completed last year. The survey did discover that the number of federal data centers grew from 432 in 1998 to 2,094 in 2010.
To investigate the industry, the Times obtained thousands of pages of local, state and federal records, some through freedom of information laws, that are kept on industrial facilities that use large amounts of energy. Copies of permits for generators and information about their emissions were obtained from environmental agencies, which helped pinpoint some data center locations and details of their operations. In addition to reviewing records from electrical utilities, the Times also visited data centers across the country and conducted hundreds of interviews with current and former employees and contractors.
Some analysts warn that as the amount of data and energy use continue to rise, companies that do not alter their practices could eventually face a shake-up in an industry that has been prone to major upheavals, including the bursting of the first Internet bubble in the late 1990s. “It’s just not sustainable,” said Mark Bramfitt, a former utility executive who now consults for the power and information technology industries. “They’re going to hit a brick wall.”
STEVE DYKES/NEW YORK TIMES ARCHIVES
A row of backup generators, inside the white housings, line the back exterior of the Facebook data center in Prineville, Ore. They’ll keep servers on in a power outage.
