AT&T says that it has largely completed the initial recovery and restoration of mobile and wireline services, after a Christmas morning explosion in which a Tennessee man blew up his recreational vehicle in front of one of its central offices, causing severe damage to buildings in the downtown area and taking down local and regional network connections for several days.
The FBI is investigating the incident, and one former agent has theorized that the 63-year-old bomber’s motivation may have been based on anti-5G conspiracy theories. He is believed to have died in the explosion.
In a series of posts, AT&T updated customers on its recovery efforts after the explosion, which caused severe damage to downtown Nashville and to AT&T’s building on 2cd Avenue, which is a local interconnect point for wireless and wireline services — but fortunately, only a few injuries. The RV was broadcasting an audible warning that it was going to explode, leading local police officers to evacuate the area.
By the evening of Christmas Day, AT&T said that it had already rerouted a significant amount of traffic that would usually be handled by that site, and brought in portable cell sites to help keep wireless connectivity. However, there were “serious logistical challenges to working in a disaster area,” the carrier noted—an area was also considered an active crime scene, as investigators worked to figure out exactly what had happened and who was responsible. Restoring power was a pressing issue, as the explosion had destroyed several on-site back-up generators on which the carrier would usually rely. On the first night, a fire in the building reignited and forced a re-evacuation, further slowing the work.
By 8:30 a.m. local time on the 26th, however, AT&T said that it had two deployable cell sites running to cover downtown Nashville as well as additional sites deployed around the city and the region, and it was still working to re-route additional services that were out. Meanwhile, its teams on the ground in Nashville had drilled access holes into the building to try to reconnect power to critical equipment via generators. The carrier reported six mobile sites were online in the Nashville area by the evening of the 26th and that power had begun to be restored by connecting a generator through the wall. Meanwhile, AT&T said it had brought back up most of the mobile network in Lexington, KY, which had been affected by the loss of the Nashville site.
By the morning of the 27th, power had been restored to four floors of the building and more than three feet of water pumped out of the building’s basement — but access to the lower areas was still limited, and teams were adding more generators and cabling to provide power, because there was damage to the building’s commercial power connections that needed to be repaired. AT&T said that it was prioritizing mobile network restoration, and that it had brought back up about 75% of the impacted mobile network sites across the region by midday on the 27th. The carrier reported at that point that it had more than 17 portable cell sites turned up to help with service, including FirstNet deployables for first responders, and another two dozen trailers of disaster recovery equipment was on the way.
By the evening of that day, AT&T reported that it had 25 deployable cell sites and 24 trailers of additional equipment deployed across the affected area, and that it had restored 96% of its wireless network, 60% of business services and 86% of consumer broadband and entertainment services. Ultimately, the carrier waived data overage fees for customers in 1,166 zip codes in Tennessee, Kentucky, Indiana, Alabama, Georgia, Illinois and Missouri.
AT&T said there was “significant damage to the building from the blast, including to the elevators, some beams/columns, and the building’s façade.” By the end of the day on the 28th, however, assessment teams had concluded that “despite the tremendous impact of the blast … the damage, including structural, is repairable” and that AT&T would be able to safely continue operating its equipment. The carrier said that its engineers were already designing permanent building repairs that could be accomplished without significant service interruptions.
“As we make the transition back to normal operations, we will continue to make repairs to the building and keep our equipment running. We will have significant resources on site as needed until the building is fully restored,” the carrier said.
The First Responders Network Authority, whose FirstNet network for first responders rides on AT&T’s infrastructure with a separate, virtualized core, said in a statement that it is already engaging with AT&T on lessons to be learned from the incident.
The Authority said that according to initial analysis, “it appears the FirstNet network infrastructure was not directly impacted by the explosion, and service continued operating on temporary battery power in the hours immediately following the event. However, because the bomb destroyed two local water mains, backup power generators were flooded and inoperable, and there was insufficient time to reroute all services before backup batteries were exhausted.”
Based on the Authority’s statement, it appears that temporary battery power fueled FirstNet services for about an hour after the blast, and FirstNet deployables arrived on-scene less than five hours after the blast — so local FirstNet services were restored within four hours after the batteries were exhausted and supporting multiple agencies that were responding.
The Authority said that it “remains in close contact with AT&T and has already initiated an in-depth review of how the network performed during this extraordinary attack on an important node of the nation’s communications infrastructure.”
“The purpose of FirstNet is to provide connectivity to first responders regardless of conditions or circumstances, and that places a heavy burden on the Authority and AT&T, the service provider, to deliver the most resilient network achievable. As we absorb the lessons learned from this attack, we will adjust our risk management and investment strategies as appropriate to deal with the changing threat environment,” said FirstNet Authority Board Chair Tip Osterthaler in a statement. “We are thankful that the only fatality from the bomb was apparently the bomber, and we are also thankful for the first responders, utility crews, and AT&T personnel who rushed to the scene on Christmas morning in order to protect lives and property and restore services.”
