Mark Bramfitt - Utilities & IT

The Real "Cloud" Story: Three Utility-like Providers

I came across this story in Quartz this morning, and found it a fascinating review of what is happening with cloud computing - starting with perhaps the best definition of what it is in the fourth paragraph:

Generically, the cloud is just a vast mass of computers connected to the internet, on which people or companies can rent processing power or data storage as they need it. It’s used for everything from hosting websites to storing archives to running massive data-crunching operations.

Besides the discussion of how Amazon has jumped the whole market, with Google in second and Microsoft a distant third, what struck me is the conclusion that compute and storage services will soon become a commodity, and yes, utility service.

That means that the three players who are building utility-scale data centers, denominated incidentally by their electric load in megawatts, are essentially building the equivalent of power plants. They're also building the data transmission and distribution system - though I think Google is clearly in the forefront on that score.

The first rule of a utility service is to drive the commodity cost as low as possible, and to monopolize the distribution market. So build the data centers cheaply, in places with cheap power, and start thinking about tying customers to your system.

As the article points out, tying up customers looks like a software/system battle now, but perhaps one of the three players (though I don't think Microsoft will make the play) will seek to buy into the distribution network - the lines that tie directly to consumers. Who wants to buy Comcast? Or maybe build their own high-speed system (Google)?

The upshot is that the leading three will be deeply entwined with utilities for service to their "power plants", and by virtue of their size will explore self generation and demand direct access to the wholesale power markets. And a huge portion of the existing loads from enterprise data centers right down to server closets will shift to the cloud, obviating the need to pursue energy efficiency as a capacity mitigation strategy.

 

Novel Use for Fuel Cells: Backup Generation

Fuel cells and solar PV get all of the attention in the data center industry as alternatives to traditional electric utility service, though there are rumblings for combined heat and power (cogeneration) installs in the background.

Part and parcel of any discussion of primary power sources (including utility service) is backup generation. Sometimes that discussion includes relying on utility service as the backup, but I imagine that the UPS and diesel generator combo has ninety-nine percent plus market share.

Now a company called CommScope is proposing the use of fuel cells as backup, rather than primary power sources for data centers, per this article in Datacenter Dynamics.

I agree with the criticism of diesel generators up to a point - they of course produce emissions of pollutants and greenhouse gases when operated and tested, but with proper maintenance they are soundly reliable. Most backup generation failures that I've heard of (many at hospitals as well as at data centers) occur due to problems in switchgear, not the gen sets themselves.

But there is another reason I think that CommScope may be grasping at straws. They are positing a sixteen hour run time, using hydrogen fuel. From my days promoting alternate fuel vehicles, hydrogen storage was always an achilles heel. Costly. Leaky. Very high pressure.

If fuel cells were to make a go of it as backup generation, I'd talk to Bloom or ClearEdge, and use natural gas service as the fuel source. 

Now utility natural gas service is not 100% reliable either, but it does have the advantage of being able to serve backup gen essentially continuously if needed.  I tried to make the case that diesel gen sets should be converted to natural gas (easy to do; 15% diesel still used as a pilot fuel for ignition) for this reason, and because emissions would be dramatically lower, but have never seen that idea take hold.

This is not to say that there aren't data center operators out there willing to try new approaches. Google put the UPS on the server in the form of a battery, and Microsoft has looked at fuel cells at the rack level.

I'd like to see a head-to-head TCO comparison of diesel gen sets and fuel cells as backups, though regardless of the outcome, an operator is looking at the most expensive power he will ever use!

 

A One-Way View: What the Data Center Industry Thinks of Utilities

I'm not sure that I'll comment as extensively as I'm tempted to do by this article from Yevgenly Sverdlick in Datacenter Dynamics - my central complaint with the article is that it doesn't present a utility perspective on any of the issues covered.

And I will say that with twenty-five years in the utility industry I am still defensive when issues like grid reliability come up. For example, I would change the opening sentence to "electrical utility grids are notoriously reliable.."

The issues of service reliability, the carbon content of power, and the opportunities for self-generation that could lead to grid independence for major data centers are all worthy of discussion, and the thoughts from the quoted industry leaders all have their merits.

But I would counter that the transmission-service power that multi-megawatt data centers get from utilities is very very reliable and a great value.

Yes, you have to make your own investments if you want a higher degree of reliability, but utilities have never been asked or required to provide that level of service. They would have to make the same kinds of investments that data centers do to provide that reliability, and the customers who want that reliability would have to pay for it. Does that sound fair? It is the basis for utility rate regulation after all.

The comments on renewables and a cleaner generation portfolio are apropos, but fail to acknowledge that getting to a cleaner, greener mix will mean higher costs. Sure, wind is pretty cheap, but it has to be offset with costs for generation or storage to make up for intermittent generation. 

And fuel cells? The mumbled truth I keep hearing is that they provide power at nine cents a kWh after tax and incentive subsidies, and with the historically low gas prices we see today. And that may not include the standby charges that the utility will assess to back you up.

Large data centers in California, paying perhaps the second highest rates in the country, see average rates of under nine cents per kWh. You bring your backup systems and factor in those costs. But please remember that the utility pays taxes, and franchise fees, and provides the money for self generation incentive programs. No tax incentives in this equation.

I said I wasn't going to comment extensively (and I could go on), and now I've made myself a liar! It is a shame though how far apart the worlds of power utilities and the data center industry remain, after a solid decade of staggering data center development in terms of both scale and scope.

I recommend this article as key reading for utility managers seeking to understand the data center market - it certainly covers that side of the story well.

Know the Retail Colocation Market

I can name a half-dozen utilities that continue to struggle with data center market assessments. That's a broad term, but one key element is determining the existing load that data centers represent in a given utility service area.

There is no question that making a qualified estimate can be done if a utility is willing to accept some reasonable uncertainty, and I will describe how I'd go about it in broad terms later. But one key element is to find the "known knowns" of the market - the large, stand-alone facilities that utilities can cull straight from an analysis of their customer records.

These facilities are "utility scale" in that they are all pulling loads of say five or more megawatts, and they are almost always self-contained rather than part of some other facility like a high rise office building, office park, or other campus.

There are three primary submarkets to look for: owner-operators, wholesale colocation facilities, and retail colo operators.

The owner-operators are the Googles, Amazons, and Microsofts, along with some financial and governmental institutions. The wholesale colos are Digital Realty Trust, Dupont Fabros, and others.

Lastly there are the retail colos, lead by Equinix and Savvis (now Century Link).

The difficult part about the colo market is that it is very fragmented - one vendor focused on this market segment said they had a list of 800 companies in the market, which means that there must be several thousand facilities out there.

This market analysis of the retail colo sector by Synergy Research Group, reported by Data Center Dynamics, gives a great snapshot. And although seven operators make up half of the market, it's pretty clear that there is still a lot of fragmentation.

Nevertheless, if I wanted to estimate data center load in a given region, one place to start for a utility is to run a search for the big names in the utility-scale segment, including the "big nine" retail colo operators noted in this report.

Customer-Side Energy Storage: An Opportunity for UPS Companies

My recent snark regarding a recent Atlantic article aside, there clearly is a growing market for electric storage both on the utility and customer side, certainly in California and likely in other high-cost markets as well.

Utilities and independent power generators are looking at storage to help balance the relatively volatile production from wind and solar plants, which because of regulatory rulings have upended the traditional loading order in some states. (The solar and wind power must be taken first, and traditional base-load plants are now expected to meet shifting demands.)

And utilities can implement storage to solve localized capacity shortfalls or peaking problems - right down at the distribution circuit level - helping to avoid capital investments that won't generate financial recovery through rates.

Customers will be brought along for the ride too. Rates will increasingly reflect the cost to serve, including punishingly high peak pricing. And if customers want to take on the onus of mitigating their peak loads through storage, they should be offered an equivalent "subsidy" that is comparable to what the utilities would otherwise have had to invest to meet that load.

As always, I want to trace this all back to data centers, which have the great advantage of an unerringly predictable load. Most data centers, whether a legacy facility or a new, highly efficient design, have load factors of slightly less than 100%. I know what the peak demand will be, because it's the same any hour of the year.

So, could data center operators play the energy storage game? Of course they could, maybe even using the banks and banks of batteries they already have on site in their UPS rooms. Except...

...that most UPS banks are sized to take a data center through a power outage only long enough to get the generators started and synced in. That's usually five minutes or less, so the UPS capacity really isn't enough to warrant a storage scheme. 

That, and data center operators are not about to discharge their UPS systems, making them vulnerable to down time in the event of an outage, to avoid peak power charges.

Finally, any data center that is a megawatt or larger is getting industrial utility rates (the lowest rates that utilities offer), so avoiding the peak power charges would be the least economically viable of any class of customers. (The customer-side storage companies should look to medium-size commercial customers for the best returns.)

So, my best and brightest take away is that the UPS companies have an opportunity to modify their products for the customer-side storage market. (Perhaps lead-acid battery systems could beat the lithium ion equipment that is getting all of the press right now.)

I'll accept a one-tenth of one percent commission for the idea!

Customer-Side Energy Storage: The Demise of Utilities?!

I had a good exasperated chuckle at this breathless account of customer-side energy storage that appeared in the Atlantic last week as part of a broader series on "The Great Energy Shift".

The report starts with the ominous warning that "trouble lies ahead" for utilities because customers are going to start installing battery storage systems that help them avoid peak pricing rates.

This is a bit of a facile observation given that utilities are increasingly changing their rate structures to reflect the true costs of providing service, particularly at peak times, that they provide incentives for load management, are experimenting with utility-side energy storage systems, and are in some cases set to provide incentives for customers to do the same.

In what way does this constitute "trouble"?

I appreciate that a hotel operator may have difficulty implementing load management strategies. I mean, turning off air conditioning in the summer in a San Francisco hotel isn't doable, not because guests who pay $300 a night won't tolerate it, but because there just isn't that much air conditioning load when it's only 62 degrees outside!

And installing an energy storage system is not "pulling the plug on the utility", it is load management, pure and simple.  And, demand rates have not risen thirty percent in California over the past three years - customers are now seeing peak usage charges for fifty or so hours out of the year.

It makes sense for the utility and/or their customers to manage that peak load. It drives the capacity ratings of the distribution system up, and really kills on the generation side, where marginal costs can spike to well over a dollar a kWh during peak periods.

I'll credit author Todd Woody with at least providing some context around energy storage, including noting that California utilities have been authorised to install 1350 MW of storage by 2020 (the costs of which will be reflected in customer rates, by the way), and that sixty percent of the costs of the system installed at the Intercontinental were covered by incentives (which will also be paid out of rates).

And yes, the system is financed by the maker of the system (Stem), though that is hardly novel, as there isn't a PV array that isn't financed today. (Stem also indicates that they sell utility-side storage systems.)

But I have to say, given all of the regulatory impetus behing real-time utility pricing, orders to install utility-side storage, and subsidies for customer-side storage, I just don't see that the "traditional utility business model, which for over 100 years ago has served its purpose well, has come to an end."

If anything, the utility business model, enabled by the regulatory process, makes the energy storage market viable.

Utility Cost of Ownership Models for New Service

I had an interesting chat with an East Coast utility client about what is essentially risk management when providing new service to data center facilities or office complexes that may include significant data center space.

The fundamental issue is that the utility is asked by the developer of the new facility to install the equipment that will serve the projected load, though there is often no assurance that the load will materialize. If it doesn't, there is wire in the ground, "copper on the pad" (transformer capacity), and electric capacity all the way back to the substation that doesn't generate revenue.

The results are at least two-fold. The utility now has assets on the ground that don't generate revenue, which means that rates for existing customers have to go up to cover the poor asset utilisation.  The second is regulatory - the utility may get a rate base dis-allowance ruling from their regulator if these situations get out of hand.

If you want a case study of this problem, look to Pacific Gas and Electric and the dot com boom of the late nineties, when it seemed every real estate developer in the Bay Area became a data center developer.

Tens of megawatts of capacity were installed for facilities that in some cases never became data centers. (Of course many of the facilities were bought for pennies on the dollar and were eventually used many years after the bubble burst.)

Now that sounds like a cautionary tale, and it is, but there are some nuances.  In California, the investor-owned utilities have a new service rule that protects them from stranded assets. 

If you are developing a facility with a significant electric load, you sign an agreement with the utility to pay a "cost of service" fee if your electric billings fall below a calculated level. In other words, if you don't spin the meter enough to justify the capital outlay for facilities to serve you, you have to make up the difference.

This rule was largely put in place to cover commercial office developments. You build a high rise building on the if-come promise of finding tenants; the utility is not going to take on that risk with you.

The problem for PG&E during the dot com bust was that the companies that signed those agreements went belly up, so they couldn't collect the cost of service fees. (To this day, PG&E remains very wary of installing services for multi-MW loads, and typically negotiates very hard to install capacity in increments.)

I was surprised to find that my client on the East Coast is only now thinking of proposing a similar scheme to their regulator. In their case, they aren't seeing multi-MW stand-alone data center development, but office projects are asking for load allocations to serve potential data center space.

It would be wise for the regulator to agree to a cost of service guarantee scheme to protect the utility and its ratepayers from stranded assets. 

And for developers of utility-scale data centers, you may find it easier to work with your utility on a phased approach to securing electric service capacity: build your substation yard with switchgear, then install transformer capacity in five or ten megawatt increments as you build out your site. Photos of Apple's new data center in Prineville suggest they are doing just that.

New Consulting Area: Attracting DC Development

After a recent appearance at a conference in Philadelphia, I've been doing some consulting for utilities in the mid-west who are interested in attracting large data center development in their service areas.

This is a new area for me, outside of the majority of work I do regarding energy efficiency program development and delivery, but really smack dab in the middle of the data center and utility industry nexus.

What I've helped these utilities grasp is the industry trends that are driving the development of utility-scale data centers, and what the operators of these facilites are looking for from utilities.

I also offer  guidance on strategies for attracting the interest of these developers, including how to accurately portray your rates now and into the future, what to say about power availability and time to serve, positioning the carbon footprint of your generation mix, and how to offer siting guidance.

There is no question that data centers are going wherever there is cheap, and hopefully relatively clean, power, but a handful of utilities (four by my count) are doing an exemplary job of preparing to answer the questions that developers need answers for when seeking sites.

I'll weave in more commentary on the industry trends and recommended strategies in future posts. In the meantime, if you are a utility looking to attract data center load as part of your economic development program, and would like to talk, please let me know!

 

A 50 MW DC Build in Houston

To all of those utility economic development representatives who wonder whether attracting new data center builds is worthwhile, note that Data Foundry has announced a new campus in Houston that at build out will draw 50 MW.

Both Data Center Dynamics and Data Center Knowledge have reports on the planned collocation development, but neither provides any information on what energy rate Data Foundry will be paying, or whether any utility construction credits or tax incentives are in the mix.

Texas is the bastion of deregulated utility markets, so in Houston you pay CenterPoint Energy for transmission and distribution services, but are free to select your energy provider (with TXU and Reliant the market leaders). Knocking around websites I found small commercial rate quotes as low as 6.5 cents per kWh, but I expect Data Foundry will pay well south of that rate.

As far as tax breaks go, the New York Times reports that Texas is the nationwide leader in corporate tax subsidies, so Data Foundry may be getting some breaks, like property tax forbearance. But as I’ve noted before, the most tempting tax break is avoiding sales taxes on IT equipment purchases – a benefit that would flow directly to Data Foundry’s tenants.

Utilities: Microsoft Is Coming...For Your Generation Plants?

The DatacenterDynamics Focus article titled Microsoft: Building Out 'At A Scale No-One Has Seen' is a must read for utilities, because it captures the intensity of the potential growth rate of utility-scale data centers, thereby informing the challenges utilities will have in serving this growth.

You'll read about how operators of utility-scale data centers have dramatically improved the design and operations of their facilities in terms of energy efficiency over the past decade or so, and even some prognostications about where those designs might go in the future. ARM-based servers that require no cooling support, fail over designs so that UPS and backup generators are mooted, etc.

But that trend doesn't make the growth rate go away. Microsoft's Christian Belady, general manager of data center services, says his team is thinking about how to scale out their data center fleet by 10x.

Why is that needed? To support two hundred on-line services like Bing and Office 365. And of course there is the Xbox, where I've seen a report that Microsoft will need to grow from 15,000 support servers to maybe twenty times that. Another source told me that Microsoft may be deploying as many as 200,000 servers per year.

What does this mean for utilities? In my view, this growth (which will also be coming from Google, Amazon, and even IBM, which is a bit of a late entry into the cloud services game but is coming on strong) will affect every utility in the country, not just those that can serve 40 MW loads.

That's because a portion of cloud services are latency-dependent, particularly content delivery. Microsoft has shown how to drop a modular data center with a load of a MW or two on a greenfield site and have it operational in ten weeks - which means the utility had to be incredibly resourceful in planning and building infrastructure.

And what other trend is Microsoft considering? Locating "data plants" right next to generating stations. The idea is that it is more efficient to move data through fiber optic lines than it is to move electrons through power lines, but I don't think that is the primary motivation here.

For the big players in this market, it must be becoming increasingly clear that getting power from utilities that meets all of their requirements (cheap, clean, available, quick) is going to get even more difficult in the future. Dropping a prefab data center on a power plant site run by a merchant generator could meet at least three of those requirements, and may be the true essence of Microsoft's thought process.