Mark Bramfitt - Utilities & IT

More Evidence of "Expanding Existing Sites" Trend

A bit of leftover news from last month, but this report in Data Center Dynamics reinforces the observation that some utility-scale data center developers are expanding existing sites rather than seeking new locations.

In this case it is Microsoft announcing an expansion to their facility in Quincy WA, following an announced expansion in West Des Moines IA.

The Quincy expansion is notable to me for two reasons. First, I had the impression that the available cheap power in Quincy (just over three cents per kWh) had been spoken for, because there hasn't been any data center development announcements for the town in some time.

If Microsoft is getting more clean hydro power in the three cent price range, that is really a coup in terms of keeping their power purchase portfolio clean and cheap.

Second, why  a 200-acre site? I have surmised that larger sites are the new paradigm so that utility scale DC operators can explore self generation (solar PV) and expand in the future. I can't see Microsoft pursuing solar with three cent hydropower as the alternative, so 200 acres might suggest a long term plan of expansion. 

Then again, $11 million for land is nothing in the scheme of things when you are developing data centers (the West Des Moines expansion project has a minimum price tag of $667 million)!

Server Room EE Program Guidance for Pacific Northwest Utilities

I recently participated in an effort sponsored by the Bonneville Power Administration to identify leading energy efficiency program measures suitable for delivery to the server room and small data center market in the Pacific Northwest. The overall project umbrella is named the Energy Efficiency Emerging Technologies, or E3 program.

The collaborative project was lead by the Wahington State University Extension Energy Program, pulling together a pretty heady roster of utility and industry participants.

The process included an initial identification of around fifty energy efficiency measures, which were whittled down to ten or so. After presentations on each of the technologies, the team chose a set of measures to pass along to BPA.

Narrowing down the measures to those that are suitable for server rooms and closets wasn't too difficult, but trying to score the measures based on their suitability for a prescriptive rebate program approach I think proved difficult for the team.

(A prescriptive rebate program offers a set financial incentive for qualified products or services - like offering $150 per server removed during a virtualization/consolidation project.)

However difficult the process might have been, I heartily agree with at least three of the four recommended measures: server virtualization, solid state drives, ongoing commissioning of economizers, and airflow management.

I continue to recommend that utilities offer rebate programs for server virtualization (really, for consolidation, and I prefer using virtualization/consolidation as the moniker) for the small data center market, as I'm convinced that market penetration there is still low. The measure captures the removal of unused but still deployed servers, the energy efficiency gains from the latest model servers, and the reduction of servers deployed through virtualization.

The solid state drive measure will need research work to develop program parameters, but I think it has lots of potential across all classes of data centers.

Commissioning of existing economizers was not a measure I supported, simply because most server rooms, and indeed enterprise data centers, don't have economizers. (The group passed on recommending an economizer retrofit measure due to installation difficulties and the looming ASHRAE regulations that would appear to mandate the installation of economizers in all new construction and for any data center upgrade projects).

Lastly, some utilities have had great success with what I call an airflow management retro-commissioning program, or "tune up". The program  pays for calculated energy savings based on on-site measurements before and after airflow management upgrades are installed, so it's not really a prescriptive program at all. Also, it is limited to server rooms and localised data centers with dedicated cooling systems, so addresses only larger facilities.

I would hope to see BPA and its member utilities take these recommendations to heart and deliver solid, consistent programs in the Northwest - there is scads of potential in this market and utilities can play a vital role in moving the market with these and other efficiency measures.

The Economist: DC Power

I love slogging through the weekly The Economist, even though I rarely make it through one before another arrives. You simply can't beat it for world affairs coverage, and the business and finance sections are awesome.

So too is the science and technology coverage, though this report claiming that the USB cable will lead to the wholesale conversion of our building electrical systems to direct current is rather over the top.

Of course in every report on DC power systems (certainly in the ones that appear from time to time about DC distribution in data centers) we have to refer to the Tesla/Edison rivalry.

But new this time around, we're told that the USB PD standard will allow us all to not only have in-home or in-office grids for all of our electronic equipment, but will also extend into multi-building smart grids using batteries charged off-peak and solar PV.

The report also mentions that data centres, "with their huge,humming arrays of servers, are already using DC circuits". I spelled "centres" the way I did because the only DC power distribution systems in data centers that I know of are in Europe, and it certainly is a very small part of the market. (Perhaps the DC side of the power supply in a server counts as a "circuit".)

My other quibble, no, lets call it something stronger, like exception, is the statement that "low-voltage DC power is cheap, safe, and green." Safe, sure, but cheap and green? Every DC system I've seen, including microgrid concepts, include connections to the good old AC grid. Where that power comes from determines cost and environmental impact. 

I can soundly recommend that utilities offer new construction and/or retrofit incentive programs for data centers for the installation of DC distribution, but I predict that there won't be any takers anytime soon. Look, if the leaders in the design, specification, and operation of data centers, who specify their own IT equipment, and who have huge buying power, aren't doing DC, it's simply not coming soon.

The USB cable standard is indeed a wonderful thing (thank you again, Intel), but I predict it won't lead to all that much when it comes to power distribution beyond electronic equipment and control systems, and it faces a challenge of its own in the form of wireless technology.

Utilities: How To Quote Rates for Utility-Scale Data Centers

In my work with utilities who are trying to attract utility-scale data center development to their service area, I make it clear that power price is the ultimate differentiator, with availability/time to serve and options for clean power secondary criteria.

The economic development representatives I've met with seem to grasp the concept pretty well, but when you test them for their power price "story", the tale is usually pretty hazy!

Let's start with the answer to this question: "What is your power price for a multi-MW data center?".

Most utilities I've worked with answer that question with "X.X cents per kWh", but in fact they are quoting an average industrial cost that ignores several key features of loads of this type.

Where does the utility rep get their price per kWh quote? Likely from the Rates Department (or equivalent), where an analyst takes all of the revenue from a given class of customers and divides it by the total kWh usage of the class.

So, for multi-MW loads, you quote the "average industrial rate" - the total revenue collected from all industrial customers (generally classified as those with peak loads of 1 MW or greater) divided by the energy use by those customers.

There are however pretty wide disparities in the rates charged to those customers, with load factor/shape and service level chief among them.

Customers with "peaky" load profiles, low load factors, and who take service at secondary or primary voltages pay lots more than customers with flat load profiles/high load factors who take service at transmission voltage.

Here's what I would do to derive a more accurate power cost quote for utility-scale data centers: I'd take a 1 MW load, assume a unity load factor, and populate a spreadsheet with 1 MW of demand and 1 MWhr of usage across 8760 hours per year.

Then I'd take my rate schedule, and I'd plot these charges in the spreadsheet:

• Customer Charge (Monthly)
• Meter Charge (Monthly)
• Power Factor Adjustment (Applied Monthly)
• Peak Demand Charge (Monthly)
• Maximum Demand Charge (Monthly, set by highest demand in previous 12 months)
• Summer On-Peak Energy Charge
• Summer Partial-Peak Energy Charge
• Summer Off-Peak Energy Charge
• Winter Partial Peak Energy Charge
• Winter Off-Peak Energy Charge
• (Set all demand and energy charges based on service voltage of secondary/primary/transmission)

If your rate structure mirrors all or many of these features, I bet the average rate you derive from this analysis is lower than your "average industrial class rate".

Now, to complicate matters for some utilities, they now have peak-period pricing reflecting actual market rates, moving down the road to real-time pricing. At the other extreme, you might find some utilities that have very simple industrial rates, though a straight per-kWh rate or even declining block rates are relics of the past for most of us.

I'll cover another aspect of the power price "story" that utilities need to be able to tell when courting utility-scale data center developers, but quoting your rates accurately for the type of load you are attracting is key.

You need the opening of your story to be: "For your multi-megawatt load, assuming you take transmission voltage service and have a unity load factor, your average rate will be X.X cents per kWh (quite a bit lower than the average rate for our customers in the industrial class)."

 

Bloom Energy Not Sole Player for Fuel Cells in DCs

Bloom Energy gets the lions share of press coverage for fuel cell installations at data centers, but ClearEdge Power has moved from their traditional residential market base into DCs with new, larger units.

ClearEdge noted the data center opportunity several years ago, but their fuel cell unit was perhaps too small to attract attention. At 5 kW in a package about the size of a rack, many units would need to be deployed to reach a scale worthy of consideration for a data center.

Now ClearEdge has a 200 and 400 kW stack, and has installed the latter at First National Bank of Omaha. Verizon has installed units also.

The trade press reports (here and here) don't cover much ground beyond the press release. For example, there is no discussion of pricing of the units or the generated power. I'd also like to know whether power delivery and conditioning equipment, namely backup generators and UPS units were obviated by the fuel cell installations.

I do think that ClearEdge may be on to something that perhaps Bloom is aware of too. Installing fuel cells at existing data centers that require electric capacity upgrades, and that receive utility service at commercial rather than industrial rates, has got to be the prime market, despite some installations at new utility-scale data centers.

If a customer can add power to their existing data center, without additional UPS and generator investments, and get power at something lower than their average rate (which can be twelve or fourteen cents per kWh on the coasts), that may be a winner.

 

EPA Utility Data Center EE Program Guide Webinar Thursday

Our bi-monthly webinar providing an overview of the EPA's "Understanding and Designing Energy Efficiency Programs For Data Centers" is this Thursday at 1 pm EST.

Check here at the EPA's web site to register for the webinar, and to get a copy of the guide.

As I mentioned on the AESP webinar two weeks ago, this is a chance to get a rather more extended overview of the guide given that we have an hour instead of just twenty minutes.

Recommended Blog: James Hamilton of AWS

I've just spent a less-than-idle few hours learning quite a bit by reading James Hamilton's blog, which I am recommending!

Amazon seems strangely absent from the discussions of the utility-scale data center industry, though of course they have to be included in any major operator list. I simply haven't heard much from that at the conferences I attend or in the trade press I watch (perhaps my choice of venues and literature need to change!).

And although James notes that the postings on his blog are his own and don't reflect official thinking of AWS, he certainly has great material on data center efficiency, power procurement, and technology for utility-scale operations.

The technology posts can get treacherously deep for my humble mechanical engineering mind, but there is great stuff here even if you're focused primarily on the utility/DC industry nexus.

Server Component Corrosion Rears Head in Polluted Environments

An interesting report today in Data Center Knowledge  that relates how Intel is finding server component corrosion beyond expected levels in equipment deployed in regions with very poor air quality - namely India and China.

Corrosion and dust accumulation were always the reasons cited for building data centers with no use of outside air for cooling. The Lawrence Berkeley National Lab did research to dispel this concern quite a few years back.

I still use a slide of their findings in my customer education deck, showing that the air outside of a facility in Fremont California was cleaner than any standard for computing equipment - though the only standard they could find was an old IBM spec. That spec was orders of magnitude higher than the requirements for healthy air from the EPA.

The real upshot of the research was that outside air, filtered to commercial office building standards, was perfectly suitable for supply to data centers.

Given the horrific reports of the air quality in China (I haven't seen much reporting on India), with PM 2.5 levels hundreds of times greater than US EPA standards, I would guess that data centers there would have significant filtering issues.

But I hope that this report does not raise the specter of declining equipment life from the use of outside air cooling back to life. 

And for extreme conditions where air quality is so very poor, liquid immersion cooling begins to look like a very promising technology!

Part II: eBay's Natural Gas-Fired Data Center in Utah

There was one more aspect to the power scheme of eBay's new data center in Utah that intrigued me: an agreement to buy power from an independent producer "over the fence" rather than through a utility wheeling agreement.

There is evidently a gas pipeline compressor station proximate to the data center, and a recovered-energy generation plant will be built there in the next eighteen months by Ormat Technologies. The plant will have a projected output of 5 MW, or a little more than a quarter of the eventual projected load of the data center.

I initially thought the arrangement strange: why wouldn't the power simply be used at the compressor station? It turns out that most compressor stations are driven by gas turbines, which put out high grade exhaust heat which is perfect for recapture, but the compressor station electric load is minimal. Usually the power generated would be sold back to the utility, though that entails building connection facilities which may be expensive depending on the plant location.

eBay has entered into a 20-year power purchase deal with Ormat, and is avoiding the grid connection only because Utah regulators allowed it. 

Now I know to the outside world these "over the fence" arrangements should be allowable, as should a wide-open power purchase market, with utilities responsible only for building, operating, and maintaining the grid. But I've pointed out before that deregulation has risks, and those have to be prudently considered by regulators and advocates.

My takeaway for utility-scale data center operators is: don't expect to collocate your facilities with power plants to avoid the costs of being connected to the grid without a regulatory battle, and indeed consider the benefits of grid connection before seeking that pathway. The industry has every right to pursue regulatory changes to meet its needs, but careful consideration is in order.

Part 1: eBay's Gas-Fired Data Center in Utah

Yevgeniy Sverdlik at Datacenter Dynamics has a good report that provides some more detail on the novel data center project that eBay has undertaken in Utah.

Some of the facility features are quite interesting - the use of both modular/container and traditional data center spaces, for example.

But of course it's the power equation that fascinates me, and though I don't agree with DCD that this is "glimpse into the future", it certainly poses interesting questions!

First off, the center relies on natural gas fuel cells for primary power, with the utility as backup. The claim is that low natural gas prices, combined with on site conversion of the natural gas to power at a better heat rate than a utility (and without the modest transmission and distribution losses), yield a cost-effective alternative to utility power.

I don't believe that that is at all correct, even in areas with high power prices.

I would hazard a guess that fuel cell power, even with Cheap Frackin' Gas (trademark pending!), runs at maybe just under ten cents a kWh at this scale. And remember, if they are using the grid as backup, the utility will be charging for that, and backup service is not cheap. California utility rates for any customer over a MW are right at 8 and a half cents per kWh.

The connection for me is that eBay can grow the facility in one megawatt increments using fuel cells, and they are also obviating the costs of UPS and back-up generators, which is no small capital cost reduction.

As to the environmental impacts, eBay will be able to claim essentially zero emissions (watch out for any gas leaks though!) but the energy source is of course not renewable. Putting that up against other renewable/clean options isn't a win but it's definitely a place or show.

I see that Microsoft is now looking into fuel cells too, but I'm guessing the scalability advantage, the stripped-down power delivery and conditioning scheme, and the environmental advantages are the primary drivers, not cost savings.