Mark Bramfitt - Utilities & IT

NYT DC Series: Riposte to Forbes Critique

The second of several posts commenting on the New York Times series of articles on data centers (here and here).

The two articles (so far; there may be more!) written by James Glanz certainly generated a lot of attention - they were among the most popular on the Times web site after publication, and generated hundreds of comments each from readers.

And of course that's not the only outlet for comment: see Rich Miller's piece on the Microsoft/Quincy article in Data Center Knowledge, and a passel of stories over at Data Center Dynamics (article here, and stinging commentary here and here).

But I'd like to take a moment to comment on this piece by Dan Woods, a contributor to Forbes.

Mr. Woods starts by saying that the opening article in the series conflates "internet" data centers with "enterprise" ones, painting the internet as culpable for the energy impacts rather than all data center operators. I suppose that's well and good, and I agree that the top dozen or two operators are pretty focused on efficiency (though I wouldn't call many of them "internet" companies).

Next we have a defense of low IT equipment utilization rates, with the quip that it would be "excellent" if someone would say what a high enough utilization rate would be. Here's my rejoinder: a lot bloody higher.

The highest utilization rate I've ever heard was from a search company (not for attribution) who said they were running in the low thirties. A major e-tailer claimed mid twenties. And yes, enterprise users are in that oft-cited 6 to 14 % range.

Mr. Woods compares these rates with the utilization of roads, telecom networks, and even newspapers. My comparison set would include any other electric appliance category, where energy use is zero when at idle. Yes virtualization can improve utilization, but implementation of true power management technology is dismally low in data centers of all stripes. Tackling this will I think be the next major area of focus for the industry.

Lastly, it is only very recently that the data center industry has become more open about sharing best practices related to energy efficiency, and certainly Facebook and their Open Compute Initiative deserve quite a bit of credit for that.

But the major players do continue to consider their operating practices to be a competetive advantage, and remain reluctant to report their data center locations, energy use, and carbon impacts. That's understandable in light of the criticism they're facing from environmental NGOs, and the undisputed fact that in their pursuit of cheap power, they are often buying from the dirtiest sources available.

First of Several: NYT Articles on Data Centers - Back Up Generators

Thought I'd comment on the two prominent articles in the New York Times on data centers (here and here) in a series of posts rather than one long one. 

The second article by James Glanz in the Times focused some attention on the pollution produced by backup diesel generators at data centers, and certainly criticized those occasions when generators are used beyond testing or to pickup load when utility service is down.

There is no question that backup diesel generators, whether used for data centers or any other critical load, are pretty serious pollution sources, which is why they are strictly regulated by local or statewide air quality agencies.

The regulators typically allow very limited testing hours for generators, and assume that additional run time for service outages will be quite limited. In the Bay Area, generator owners are allowed twenty hours of testing a year. In exchange for these limitations, owners do not have to install expensive pollution control measures.

Even with these limitations, air boards have had to wrestle with the concentration of diesel generators needed for utility-scale data centers. And when there is a concentration of data centers in a given area, this concern is obviously magnified. (Think of the example cited in the article, Quincy WA, as well as the City of Santa Clara's Space Park Drive area, and Digital Realty's data center campus in North Carolina.)

I'd like to make two points on this issue.

First, data center operators should limit diesel generator use to testing and emergency backup only, and utilities should not promote generator use for demand response programs absent the installation of the best available pollution control measures on the generators.

It's hard to discern what was really happening with generator use at the Microsoft data center in Quincy; I could guess at only two reasons for it and only one of those comes close to a "no foul" situation.

If data center operators are encouraged by utility programs or policies to run generators outside of testing and emergencies, then they should be obligated to work both with their customers and air quality regulators to ensure that suitable pollution control improvements are installed. Substituting the operation of distributed diesel generators with limited emissions controls for the use of peak power generators (that are usually quite dirty too) may make some economic sense, bu is probably a loser when it comes to overall emissions output.

Second, it's time for data center operators with multi-MW of generator capacity to consider cleaner options. (If you don't agree, stand in a yard full of 2 MW generators when they fire up at once - I won't pay for your dry cleaning bill.)

Diesel generators will operate on natural gas with a fifteen percent diesel portion, significantly reducing emissions. I know data center operators want to have sufficient fuel on site to guarantee a given run time, and don't consider natural gas from a local utility a reliable enough source. So what we need from the generator companies is a switchable unit that can rely on natural gas when available, then switch over to 100% diesel if needed.

I think regulators are going to force this issue, especially in areas with high concentrations of generators.

A utility/regulator/data center operator deal that works around these issues could yield lower emissions, the ability to run generators for demand response, and easier permitting for new utility scale facilities. 

Koomey's Report For New York Times on DC Energy Use

A day late in commenting, but I found this report from the New York Times on data center energy use fascinating in many ways, not the least of which is that the report prepared by Jonathon Koomey was in fact sponsored by the newspaper.

Jonathon was asked to revisit his influential 2007 report from the E.P.A that predicted a doubling of server and data center energy use over the 2005 to 2010 time period, reaching a demand of 12 gigawatts this year. (I know: these reports conflate energy use and demand to a degree that can confuse an engineer, let alone a layman.)

His report concludes that data center load in the US has only hit 17 GW because of a combination of the economic downturn and a new focus on energy efficiency. (My guess is that the economic downturn is by far the prevalent factor.)

This droop in the growth curve has been predicted by others - I know Christian Belady at Microsoft predicted it as well. My question is whether the focus on energy efficiency has moved down into the enterprise data center market or is rather more contained in the utility scale operator group. If it hasn't, then the industry has a chance to deliver energy efficiency gains at the same time as the economy (hopefully) picks back up.

I noticed too that Google is mentioned in the article as being responsible for less than one percent of the energy use in the sector, attributing the low number to their intense focus on efficiency. That puts their demand at something like 150 MW. For perspective, there are a handful of data centers in the US that have loads in the 100 to 200 MW range by themselves.

 

Consolidation and Actual Data Center Closures: Two Examples

While at Pacific Gas and Electric Company, I often claimed that we had never seen a customer abandon data center infrastructure - rebuild and reconfigure, or sell, yes, but never decommission the infrastructure investment required for this power hungry use.

Two recent developments may belie that kind of claim.  First, the federal government appears to be hastening it's move to consolidate data centers, as reported in today's New York Times, and that includes outright closure of data centers.  I expect that most of these will actually be in the server room/server closet category, but a large data center operated by the Homeland Security Agency is slated for closure as well.

To some extent the federal story is about moving to cloud infrastructure as a means of containing infrastructure sprawl, and perhaps gaining efficiencies from higher utilization and newer, more efficient infrastructure.

I've also heard that a major financial institution is closing two large legacy data centers in the Bay Area, consolidating the workloads in facilities in the south where the company is headquartered. Given the large number of mergers and buyouts in this sector, I suppose it is no surprise that eventually back office operations like IT are eventually consolidated.

The two centers in the Bay Area, while quite large in size and electric load, were hampered by low power capacity - they could support something like twenty-five or thirty watts per square foot. In today's world, that leaves you with highly-loaded racks with lots of open floorspace, which prevents efficient cooling system operation.

If you can't bring more power into these legacy centers, or if the costs of adding cooling and power infrastructure to the existing space is high, it's worth building new. I still wonder whether new tenants at these facilities will make use of the data center space, perhaps by reducing the floor area or just living with the low available power densities.

On Politics: Campaign Noted in Local Press

While I will continue to focus my blog postings on my professional consultancy, I will occasionally provide updates on my incipient campaign for First District Supervisor in Sonoma County.

With a mention in a story in the Santa Rosa Press Democrat this morning, I would have to say that we've at least "soft-launched" an announcement regarding my interest in running. 

The primary election of course is more than a year away (June 2012), but stay tuned for updates here, as well as an announcement of my campaign site.

All wishes of good luck are welcome!

My Absence - Explained!

To regular readers and my patient clients, my apologies for the lack of new material over the past three weeks or so.

Elated to report that my wife Nicole and I were in Richmond Virginia for most of that time, greeting our newborn adopted son into the world.

I expect I'll post perhaps more frequently than I have done in the past, but some may be written in the middle of the night!

Lots to catch up on, so check back in over the next few days.

Jay Fry Interview

My decision to leave PG&E, or at least my announcement in October, caused rather more interest and speculation than I had anticipated.

Fortunately, Jay Fry of CA, who I got to know when he worked for start up Cassatt, offered me the opportunity of an interview on his blog, where I could explain my future plans.