So how is the communications infrastructure holding up now that Hurricane/Superstorm Sandy has blown through the Tri-State area? Not so good. The image you’re looking at is of the flooded ground floor of Verizon Headquarters at 140 West St. in lower Manhattan. I haven’t been downtown today, but if that’s an indicator of what’s going on down in the communications-dense lower portion of Manhattan, then there are a lot of CIOs with big waterlogged headaches today.
Let’s start with the wireless networks. Anecdotally, I’ve been hearing about all sorts of difficulties with voice calling today. For the first half of the day, my Verizon iPhone didn’t display the 3G icon at the top of the screen, and the few voice calls I tried to make took longer to connect than usual if they connected at all, which about half of them didn’t. I also heard that people calling me were hearing “fast busy” signals indicating an overloaded network.
That’s not surprising because the storm zone is heavy Verizon territory. In a statement to Reuters, Verizon said it was still trying to get a full picture of the damage to its network. Sprint and T-Mobile are pretty much in the same boat, and customers of all three are seeing service disruptions. So for the time being my advice about texting and not calling remains in force.
Then there were the land lines and Internet services. Again, this is mostly Verizon territory. The company said in a statement that several of its central offices in Manhattan, Queens and Long Island had been hit with flooding and had also lost power. Normally there’s backup power, but generators don’t exactly respond well to water. So they’ve been running on battery power. The problem is when the batteries die, the equipment inside the hubs gets damaged and so Verizon has had to power it all down, thus hurting its capacity for service. This applies to things like FiOS, its DSL Internet services and old-school telephone service. A fellow named Jack is doing yeoman’s work on Verizon’s customer support Twitter account trying to help customers out.
On Long Island, where most of the people rely on Cablevision for their Internet, about 90 percent of residents have lost power. That’s affecting access to the Internet and phones on Cablevision’s Optimum Online service. Time-Warner Cable is suffering outages in parts of New York City and New Jersey, where people have also lost power. The company says that as power comes back, they’ll identify people whose service has been disrupted.
Personally I can report that RCN, the regional cable company, managed to keep its TV service running, but Internet service was disrupted for about 12 hours in Manhattan. Some lost TV service too, despite the power staying on in most of Upper Manhattan.
And the woes weren’t limited to consumers and their cable modems. This may come as a surprise, but data centers don’t like water very much, and lower Manhattan flooded pretty badly. One data center operated by a hosting outfit called Datagram, whose most high-profile customer appears to be Gawker and its network of sites, saw its lower Manhattan facility flooded. Normally backup power would have kicked in but according to a terse statement, the basement where the generator’s fuel tanks are kept was totally inundated.
Here’s what Datagram said in full:
As of 5pm on October 29, 2012, Datagram had thoroughly tested its emergency systems at 33 Whitehall, NYC and was fully staffed awaiting the storm to hit Manhattan’s shores. Once ConEd lost power to Lower Manhattan, Datagram’s emergency systems kicked on maintaining power to Datagram’s datacenter. Unfortunately, within a couple hours of the storm hitting Manhattan’s shores, the building’s entire basement, which houses the building’s fuel tank pumps and sub pumps, was inundated with water taking the building generator system offline – essentially shutting down the entire building. No Datagram or customer infrastructure has been damaged by the storm.
We have been working intimately with the building’s engineers to clear the water from the building’s basement so that we may restore our emergency power systems. We are providing regular updates through our website at http://www.datagram.com.
A similar problem hit a huge co-location facility at 75 Broad Street. As Data Center Knowledge noted, flooding there destroyed critical fuel pumps needed to keep backup generators online, disrupting service to customers of hosting providers like Internap and Peer 1. An emergency letter posted as an image to Twitter warned customers that they should move their workloads elsewhere. At the time it was posted, the facility had about five to seven hours worth of fuel to keep the lights on and servers humming. It subsequently lost power before noon local time.
They weren’t alone. Renesys, a research firm that tracks the health of the Internet, posted on its corporate blog a graph showing a surge in network outages in New York and the surrounding area. (I borrowed the image from Renesys. Click to make it bigger.)
There’s also some interesting intelligence from a new site called Outages.org that has a really detailed report on conditions at 111 Eighth Ave., a.k.a. the site of Google’s New York operations.
It isn’t just a Google facility, but it is one of the most heavily connected buildings in New York. The detailed report on Outages.org lists lots of companies affected by flooding, including 8×8, the Internet phone service Voxel, cloud services and hosting company Equinix and XO Communications.
I’m keeping an eye on things and will report back on more situations as they develop. If you know of any other service disruptions involving data centers or any other service providers, drop me a line.