Mark Bramfitt - Utilities & IT

EULs for Data Center Power Equipment

A bit of utility energy efficiency program managers insider baseball, so skip if not of interest!

I collected a bit of useful data yesterday at the Technology Convergence Conference hosted by Teladata in Santa Clara yesterday in a presentation by Eaton, a big provider of power delivery and conditioning equipment for data centers.

Eaton posted a short list of the expected useful life for four classes of equipment. That's obviously useful for data center facility managers who have to anticipate equipment replacements, and even for accountants who have to set depreciation schedules. (Well maybe not for the accountants as I suspect the schedules are set by IRS policy, not reality!)

Here's the list:

UPS equipment: 12 years

Transmission switchgear: 12 - 15 years

CRAC units: 10 years

Circuit breakers: 12 years

It's a pity we don't have estimates for power distribution units and distribution transformers, and I wonder if there isn't quite a lot of variation for cooling units (how long to CRAHs last?), but there's something here for utility program managers as well as data center operators.

For the DC managers, I wonder if they really plan for equipment replacement based on these time frames. In particular, I've seen switchgear and circuit breakers that have lived several lives based on these numbers!

For utility program managers, the UPS and CRAC units figures could be a good basis for establishing Effective Useful Life (EUL) metrics for your energy efficiency programs. 

Let me explain that in layman's terms for non-program folks: If you approach a utility for an energy efficiency rebate for a UPS system that is say nine years old, they are likely to base the payment on the energy savings of a premium efficiency unit you might purchase, compared to a "standard" efficiency unit on the market today - not the difference between the unit you buy and the existing unit.

The premise is that at nine years old, you have to buy a new unit anyway, so the efficiency gain of buying a modern unit is a given. The utility will only reward you for going to a premium efficiency unit.

Now, if you replace a unit that is not close to the end of its EUL, the utility can provide more of an incentive because you are retiring the equipment early.

I'll look for more references to EULs for data center equipment out in the industry and pass them along as I uncover them.

 

Attending: Technology Convergence Conference February 18th

Happy to note that I'll be prowling the sessions at Teladata's Technology Convergence Conference next week in Santa Clara, looking out for industry trends in general and anything that hits the utility and data center industry nexus. (No speaking or panel moderation duties this time around!)

The TCC events are really a great way for me to catch up industry players, hear from vendors, and broaden my horizons a bit beyond just energy efficiency issues. I've learned a lot about other aspects of data center infrastructure - a perspective absent from most other events.

It's a great credit to Teladata to put these events since 2007. 

Check out the agenda here, and of course I'll report on interesting revelations in this space later next week.

A One-Way View: What the Data Center Industry Thinks of Utilities

I'm not sure that I'll comment as extensively as I'm tempted to do by this article from Yevgenly Sverdlick in Datacenter Dynamics - my central complaint with the article is that it doesn't present a utility perspective on any of the issues covered.

And I will say that with twenty-five years in the utility industry I am still defensive when issues like grid reliability come up. For example, I would change the opening sentence to "electrical utility grids are notoriously reliable.."

The issues of service reliability, the carbon content of power, and the opportunities for self-generation that could lead to grid independence for major data centers are all worthy of discussion, and the thoughts from the quoted industry leaders all have their merits.

But I would counter that the transmission-service power that multi-megawatt data centers get from utilities is very very reliable and a great value.

Yes, you have to make your own investments if you want a higher degree of reliability, but utilities have never been asked or required to provide that level of service. They would have to make the same kinds of investments that data centers do to provide that reliability, and the customers who want that reliability would have to pay for it. Does that sound fair? It is the basis for utility rate regulation after all.

The comments on renewables and a cleaner generation portfolio are apropos, but fail to acknowledge that getting to a cleaner, greener mix will mean higher costs. Sure, wind is pretty cheap, but it has to be offset with costs for generation or storage to make up for intermittent generation. 

And fuel cells? The mumbled truth I keep hearing is that they provide power at nine cents a kWh after tax and incentive subsidies, and with the historically low gas prices we see today. And that may not include the standby charges that the utility will assess to back you up.

Large data centers in California, paying perhaps the second highest rates in the country, see average rates of under nine cents per kWh. You bring your backup systems and factor in those costs. But please remember that the utility pays taxes, and franchise fees, and provides the money for self generation incentive programs. No tax incentives in this equation.

I said I wasn't going to comment extensively (and I could go on), and now I've made myself a liar! It is a shame though how far apart the worlds of power utilities and the data center industry remain, after a solid decade of staggering data center development in terms of both scale and scope.

I recommend this article as key reading for utility managers seeking to understand the data center market - it certainly covers that side of the story well.

NTT Building a Global Retail Colo Collosus

Another notable report on the retail colo industry from Data Center Knowledge, where NTT has completed an eighty percent buyout of RagingWire, doubling it's United States portfolio to 650,000 square feet.

RagingWire has a major data center in Sacramento, as well as Ashburn Virginia; NTT has existing colos in Silicon Valley and also in northern Virginia.

We'll see if NTT grows it's US presence through further acquisitions, through development, or both. There is lots of room for consolidation in the retail colo market, as we have nothing like the "four companies holding a combined eighty percent market share" that most industries seem to evolve to.

However NTT grows, utilities will find them a good partner for energy efficiency engagements, as they have a long history of making investments with the long term in mind, while also adopting early stage technology.

 

Wholesale and Retail Colo Markets Blurring

According to Data Center Knowledge, wholesale colocation data center center company DuPont Fabros is dipping its toes into the retail market, with cabinet level leases available in New Jersey and northern Virginia.

DFT joins Digital Realty Trust in exploring this market - Digital began offering smaller space increments, now under the moniker of Turn Key Flex, some time ago.  Digital offers modular units that are a megawatt or more of IT power capacity, not rack scale units.

DuPont's focus on New Jersey makes obvious sense to me, with reports of something of a tech boom in the New York City metro region, and of course the always hopping financial services sector. Lots of small tech (and probably financial) firms opt for cabinet level colo, so they should have success there.

Northern Virginia lives off of the government and defense sectors in Washington DC, and it was my impression that customers were therefore much larger, requiring the wholesale space that is the bread and butter for both DFT and Digital.

Colos have been a difficult market for utility energy efficiency programs, and none more so than the retail sector. Most of the EE measures for data centers require the cooperation of whoever runs the rack, from selection of equipment and technologies, virtualization, and on to airflow management. When the cabinet manager is renting from the facility owner, the split incentives tend to stymie cooperation on efficiency.

But if these market moves mean new construction, utilities can still play a big role promoting economization and premium efficiency equipment (for cooling and power distribution).

As far as economic development, retail colo gets built near potential customers - they aren't built where is land is cheap and power almost free. So, good news for utilities in established colo markets, but not opportunity for those trying to attract data center development in non-traditional areas.

Optical Data Storage Returns!

I was really intrigued to see this report from Data Center Dynamics of an announcement by Facebook at the Open Compute Summit earlier this week regarding another data storage innovation.

Facebook has designed an optical data storage array using Blu-ray disk technology, delivering high data density, very low power use, and long storage life.

I remember talking to optical storage industry people back in 2005, when they were considering whether to try and get into the data center storage market (using DVDs). The upshot was that the industry focused solely on consumer products, and of course hashing out the next optical disc standards.

The prototype rack holds 10,000 Blu-ray disks in magazines, using mechanical means to access and read the disks, very much like a tape magazine system. The array holds a petabyte of storage, and uses eighty percent less energy than Facebook's other "cold storage" innovation, which is essentially a data center-scale MAID system.

(The data center-scale MAID system is a separate room for data "cold storage", where rarely-accessed data is saved on hard drives which are then shut off. You can get the data back, but with somewhat slower access times. By housing the equipment in dedicated rooms, the environmental conditions can be set far wider than for other data center IT equipment, delivering greater energy savings.)

There is other news from the conference this week that is of interest to data center energy efficiency junkies, but Facebook deserves lots of credit over the years for exploring data storage options. They made use of rack scale MAID systems years ago, and continue to lead the way with these new schemes.

I think there are real opportunities for utility energy efficiency program offerings in the data storage space, despite some challenges. At the very least, there is an opportunity to educate customers about energy efficiency options, and I think an incentive program could be developed to support the adoption of solid state, and now optical, data storage technologies.

Know the Retail Colocation Market

I can name a half-dozen utilities that continue to struggle with data center market assessments. That's a broad term, but one key element is determining the existing load that data centers represent in a given utility service area.

There is no question that making a qualified estimate can be done if a utility is willing to accept some reasonable uncertainty, and I will describe how I'd go about it in broad terms later. But one key element is to find the "known knowns" of the market - the large, stand-alone facilities that utilities can cull straight from an analysis of their customer records.

These facilities are "utility scale" in that they are all pulling loads of say five or more megawatts, and they are almost always self-contained rather than part of some other facility like a high rise office building, office park, or other campus.

There are three primary submarkets to look for: owner-operators, wholesale colocation facilities, and retail colo operators.

The owner-operators are the Googles, Amazons, and Microsofts, along with some financial and governmental institutions. The wholesale colos are Digital Realty Trust, Dupont Fabros, and others.

Lastly there are the retail colos, lead by Equinix and Savvis (now Century Link).

The difficult part about the colo market is that it is very fragmented - one vendor focused on this market segment said they had a list of 800 companies in the market, which means that there must be several thousand facilities out there.

This market analysis of the retail colo sector by Synergy Research Group, reported by Data Center Dynamics, gives a great snapshot. And although seven operators make up half of the market, it's pretty clear that there is still a lot of fragmentation.

Nevertheless, if I wanted to estimate data center load in a given region, one place to start for a utility is to run a search for the big names in the utility-scale segment, including the "big nine" retail colo operators noted in this report.

Wholesale DC Market in Oversupply?

Of particular interest to utilities seeking to attract data center development in their service areas...

I've noted announcements from several of the largest utility-scale data center operators (like Microsoft and Apple) indicating that their build-out strategy is increasingly aimed at expanding existing sites rather than seeking new locations. That's not great news for utilities looking to attract these facilities as part of their economic development activities.

Now Data Center Dynamics reports that the outlook for the wholesale colocation market is decidedly mixed for the short term, with some apparent overcapacity. In something of an ironic twist, the report notes that some of that capacity is due to large operators moving out of colo into their own facilities.

The textbook example in my mind is Facebook, which began their life in retail colo space, migrated to wholesale, and now of course operate their own centers. This has left a hole in the Fortune Data center in San Jose, though Facebook appears to be on the hook under a long term lease.

Some big leases were signed last year, and some big increases in available space are expected in 2014, but I'd guess that the big wholesale players (Digital Realty, Dupont Fabros, etc) will be treading very carefully in considering new development.

I'm also intrigued that Microsoft signed a few leases last year, but for comparatively small blocks of capacity (all under 10 MW). I'd posit two possible explanations.

The first is that Microsoft may lease small blocks to align their growth against their own capacity, filling in near term shortfalls with the leased space. The second is that the small increments may be the best way to address service latency requirements in locations that aren't near company facilities.

I'm told Microsoft is looking to have some capacity in every city with a professional sports team, simply to meet the latency requirments for it's Xbox system!

I still think that a utility that can put together a package of low rates, capacity availability, short time to service, and a clean generation mix (or access to it) will attract attention from both owner-operators and the wholesale colo market. However the trends of expanding existing sites by the owner-operators, and short term overcapacity in the wholesale market are headwinds that have to be overcome.

Tour: "Cabinet Colo"

I'll be posting occasional "Tour" pieces that might be helpful for utility program managers and account reps who are interested in becoming more knowledgeable about the data center market. If you've been around the industry block for awhile, by all means ignore these!

It's frankly pretty rare for data center operators to allow you to photograph their facilities, so I was really pleased to get these shots of a "cabinet colo" operator in California late last year. Although they didn't expressly prohibit me from naming them, let's leave them anonymous for now.

(However, if you're looking for cabinet level colocation services, let me know and I'll refer you to this top-notch operator!)

I've called this operation a "cabinet colo" because they make their business the leasing of colocation space primarily by the cabinet. They'll rent you space for a 1U piece of equipment, and they do have some whole rooms available for sole tenants, but ninety percent of their clients rent in cabinet increments.

The first photo gives you some sense of the scale of the facility, with rows extending maybe two hundred feet broken by a central aisle. The current phase of this facility has maybe thirty or so rows and the capacity for at least that again, plus the adjunct rooms that are currently unused.

As you walk down the rows you find everything from empty cabinets, cabinets with just a few pieces of gear, and multiple cabinets that are fully utilised. The ones in the last category often feature a standard equipment configuration that is duplicated in every cabinet.

I did see an occasional cabinet filled with "desktop" servers - simply servers in a PC cabinet form factor stacked on shelves.

Working in the colo can be fairly lonely, as we see in this photo of an IT tech.

I also took a shot of this - this is how servers are shipped, in lots of five, from the OEMs.

I always remember Christian Belady of Microsoft saying that one reason to pursue modular solutions is to avoid having to open these boxes and rack the equipment into cabinets. At a Microsoft data center I visited, they had a side room where technicians racked gear into cabinets, which were then wheeled into the data center and connected.

Now many of the utility scale data center operators order "rack in a box" solutions, and of course at the extreme source fully racked modular data center units (like the shipping container solutions).

Lastly, in terms of energy efficiency, you can appreciate the challenge of cooling a colo, and certainly a cabinet colo. Although there is usually an upper limit on power density per cabinet, the facility operator has no control of what is in each rack. With a completely heterogenious load profile, it is impossible to deploy cooling solutions that limit excess air flow and supply and return mixing.

This operator intends to deploy aisle containment when the center is more fully leased, maximizing the ability to use outside air.

The Return of...Magnetic Tape Data Storage?

Every quarter the Economist runs a special technology section that never fails to enchant, and the latest version in the November 30th, 2013 edition) didn't disappoint.

A story entitled "Magnetic tape to the rescue" caught my eye, and if the purported trend to revive tape storage in the era of "big data" holds true, it may have some interesting impacts on data center energy use.

First the article notes that the growth in stored data is marching at a much faster pace than the need for compute resources, doubling every two years. Compute growth has evidently slowed from a rate of doubling every five years to something like a ten percent annual clip.

But for all the talk of solid state storage systems taking market share from spinning disks, the inexorable decline of magnetic tape systems seems to have stopped, and use of the oldest storage technology may actually begin to increase.

Mag tape systems are still used for long-term archiving of data that is very rarely accessed. Basically you store info to tape and send it off to a repository somewhere, hoping never to have a need to retrieve it.

Its an excellent medium for this use, with great data density, low cost, long life, and next to negligible energy use (not much to record it, and not much to keep the store room cool and dry). So, for storage of "cold data", it really can't be beat.

Of course there are efforts afoot to make tape storage even more attractive by increasing storage density (35 terabytes on a single cartridge!).

I do wonder if the reintroduction of tape drives in data centers might have negative consequences on overall energy use. It was always conventional wisdom that the very tight environmental requirements that are still pervasive in the industry were relics of the days of punch cards and mag tape drives - especially the very narrow tolerances for humidity.

I expect that if mag tape drives still require special environmental conditions, they'll be housed in separate rooms in efficient data center designs, so that the "factory" machines (servers, com gear, and solid state and spinning disk drives) can sweat it out on the shop floor.