Restoring Networks After Sandy

One of the unfortunate consequences of events like the Northeast “Frankenstorm” is the speed with which they're exploited for various kinds of gain. When food and water are short, vendors show up on street corners selling goods at exorbitant prices, looters rob stores of computers and that sort of thing. After the fact, many people with a policy ax of some sort will point to various things that happened as examples of tragedies and inconveniences that could have been avoided if only they'd had their way in the policy process. The National Association of Broadcasters jumped on Frankenstorm with a rather thin argument to the effect that the storm proves their networks are great for communication, despite their one-way nature. Another example of this phenomenon that struck me as particularly odd was a blog post written by Harold Feld on Wetmachine before the storm had even hit: If your cell tower loses power, be sure to thank CTIA and the D.C. Circuit. Feld argues that the FCC needs broad Title I authority over Title III cellular networks to keep them running after hurricanes strike:

As we hunker down to wait out Hurricane Sandy, some folks have noticed that if we lose power our cell phones might not provide the back up we expect. Cell towers require power, and if the back up battery is drained and local power is not yet restored then the network goes dead.

He goes on to mention a recommendation by the Katrina Commission for cell towers to have sufficient backup power to keep working for 8 to 24 hours after a loss of general power. You can't completely fault policy advocates for using disasters to further aims that are directly related to disasters - these things are “out of sight, out of mind” most of the time so each disaster is a “teachable moment.” But I'd like to examine the recommendation in light of what’s happening in New York and New Jersey to do a little exploiting of my own. Why should Feld have all the fun?

Three days after the storm, there were still widespread cellular outages in parts of New York and New Jersey, so the requirement for 8 - 24 hours of battery backup is clearly moot. Reports indicate that the networks suffered various forms of damage that would not have been mitigated by power to base stations in any case: Towers blown over by wind, flooded, and knocked down by trees don't work whether they have power or not. Similarly, some major data center/switching center locations in New York were flooded, and trees and wind damaged some overhead cables that provide towers with backhaul connections to switching centers. The overall damage to Internet connections in the Northeast took out about ten percent of the local Internet, including Gawker, Huffington Post. and Buzzfeed:

Renesys concludes that about 10 percent of the networks in the New York metropolitan area went down, which is really low given the fact that the local electric company, ConEdison, actually cut power to the lower portions of Manhattan, where a lot of those networks are based.

All in all, the Internet survived the storm a lot better than the telephone and power networks did. It did this mainly because switching centers are well built and equipped with backup power in the form of generators and tanks full of diesel to fuel them. A number of web sites with local focus are actually experiencing record traffic today, even in areas like Montclair, New Jersey, that are still without power:

Sheepshead Bites, a four-year-old independent site covering the Sheepshead Bay neighborhood of Brooklyn, also reported its highest-ever traffic day Monday on its Facebook page. And eight-year-old independent Baristanet, which covers Montclair, Bloomfield and Glen Ridge, N.J., reported traffic Tuesday was on course to be three times higher than the average 8,000 to 9,000 daily visits, notwithstanding the fact that much of Montclair is without power.

How is this possible when the FCC lacks the Title I power to require the cell towers that are carrying the traffic to these power-deprived localities to have 24 hours worth of battery backup?

It turns out that the cellular network operators have responded to the storm in a number of creative and interesting ways, as have many Internet operators. AT&T and T-Mobile combined their operations in the affected areas, allowing any customer of either network to connect to any available tower:

In the wake of Hurricane Sandy, much of lower Manhattan has completely lost electricity and cell reception. T-Mobile USA and AT&T said on Wednesday that in the affected areas of New York and New Jersey, their customers would be able to use the networks of both companies, decreasing the likelihood of failed calls.

Oddly, they did this without anyone forcing them. In addition, all the operators have been powering tower that aren't getting power from the grid with backup generators that can last much longer than the Katrina Commission recommended provided they can get diesel. And in fact the biggest problem with restoring service to these locations on an ongoing basis is the simple matter of refilling their tanks. Power loss makes it hard to pump fuel, you see.

The FCC continues to monitor the restoration efforts following Hurricane Sandy and sees improvement.

Agency Chairman Julius Genachowski says though much work remains to be done to restore communications services fully, there is continued improvement. The fuel supply for generators supplying communication networks is essential and the FCC is working with federal, state and local authorities to speed up those deliveries, he notes.

FCC field agents in New York City have been supporting FEMA and working to get fuel to a switching center that serves many communications providers. They agency has issued another STA, this time to an energy company assisting restoration operations in several states.

So the FCC’s power to order what’s needed - diesel fuel, not battery backup to towers - seems to be adequate.

The fantasy that batteries on cell towers is the key to reliability is some old-school telephone network thinking. In the old days, telephone central offices supplied power to black telephones from 48 volt batteries down the line to each house. This was designed into the telephone network mainly because it had more reach in the early days than the power grid. The cellular network and the Internet are different, which is why we have backup generators where necessary.

New Yorkers with nothing working but their mobile phones are struggling to recharge, and the FCC is also using its informal power to encourage the operators to send RVs set up as charging stations into affected areas:

AT&T is working with Mayor Bloomberg and the City of New York to quickly deploy generator-driving charging stations and RVs with charging capabilities for New Yorkers at local Food and Water Stations being set up throughout the five boroughs.  In addition, people in affected areas are invited to visit any of AT&T’s approximately 400 retail store locations now open through the Northeast to charge their devices. Details on open stores and temporary/mobile charging stations will be posted as soon as available at att.com/aboutus.

So it seems that those crying wolf for Title I authority spoke a bit too soon. Cellular network operators seem to be doing a great job of restoring service by a number of means that weren't contemplated by the geniuses on the Katrina Commission despite the fact that nobody’s holding a gun to their head. Sometimes people and businesses do the right thing simply because they're decent and/or because they understand the importance of reputation.

We don't need the courts to remind us of that. It’s going to be two weeks before the networks are fully repaired, but as far as I can tell there’s no lack of effort by the network providers. If only the power companies were half as ingenious a lot of New Yorkers and New Jerseyites would be better off right now.

[Note: This was originally posted at High Tech Forum on November 2.]