As part of our school safety security plan review checklist I want to take you back to an incident from the summer of 2012 when Murphy’s Law struck and 911 failed and ask you what would you do? On June 29, 2012, a derecho, a thunderstorm similar to a hurricane making landfall, struck Northern Virginia. There were downed trees, power lines, cell towers and other significant damage causing widespread power outages. While emergency generators were in place and most powered up properly, two failed to start beginning a cascading failure of the 911 system. Multiple system and communications glitches contributed to the failure and the system remained down for up to four days. Emergency responders scrambled to find workarounds, including re-routing calls through non-emergency lines. Murphy’s Law had struck 911 systems before during 2010 and 2011 blizzards, as well as Hurricane Irene.
School Safety Security Plan Review Checklist
Important Lessons Learned
There are important points to learn from these events, the first of which is that anything mechanical or designed by a human will fail and it will do so at the most inconvenient time imaginable. The next point is failing to prepare for and improvise around system failures often results in a larger catastrophe to deal with than otherwise would have occurred. The more complex a critical system is, the more failure points become possible and the more difficult to plan for. Perhaps the most important point is that any emergency plan or system is only as strong as its weakest link, usually involving human response to a mechanical failure.
Although the generators that failed to start had been checked just three days prior to the event, when they failed, it caused an overload, as the huge increase in calls were rerouted through fewer lines and had knocked out Verizon’s monitoring equipment. Verizon, responsible for that 911 hub, failed to notify emergency personnel of the failure for more than several hours and emergency crews scrambled to notify the public that the 911 system was unreliable. Even though some non-emergency lines worked intermittently, landlines, wireless services, including texting capability, was damaged. Emergency crews resorted to TV stations, radio stations, newspapers, blogs and social media sites to advise residents to flag down passing police officers or walk to the nearest fire station for assistance if they couldn’t get through on 911.
Security Planning-Preparing for Failures
As security professionals, planning for these failures presents significant challenges. It is impossible to anticipate all potential failures both mechanical and human. It is also frequently challenging to obtain support from administrators and non-security personnel for ideal levels of training, drills and budgetary support. Given those challenges, there are creative measures we can utilize to reduce the chaos Murphy’s Law poses.
After the incident, Verizon stated that while they had dealt with main power failures and problems with generators and other back up power resources, they had never had to deal with them all at once. As part of their plan to address the issue, they are building in further redundancies, improved monitoring, feedback and notification systems when failures occur. While each facility and security team will have unique circumstances to plan for, building in redundancy into critical systems, especially communications should be a high priority. In addition to land lines and cell phones, social media, including twitter, as well as emergency hand held radios and even ham radio are potential communications alternatives when a catastrophic failure occurs.
Diversifying emergency power sources, communication capability and other emergency response increases the odds that at least one source will function when needed, especially when staged in different areas. If one area of the facility is damaged or destroyed, having emergency equipment accessible in other areas can make the difference between an inconvenience and a catastrophe. Additionally, striving to prevent single point failures that cascade onto remaining systems is well worth the time and energy spent. This is best coordinated with facility maintenance personnel and they should be included as an important part of an emergency plan.
Developing good working relationships with emergency personnel and organizations, such as FEMA, the Red Cross and other local first responders well before a crisis occurs is an indispensable part of security planning. Often, they can offer experienced observations for resolving emergency planning that can help save time, money and effort for the unique circumstances a given area will face. These relationships also serve to ensure smoother responses from both the facility security team and the emergency personnel. Including other nearby facilities and security teams increases the strength of the network in the event of a catastrophic event.
My Field Experience
Fortunately, in spite of the various hurricanes and other disasters I worked in, I never had the misfortune of having such a catastrophic failure of the 911 system. Generators worked, sometimes communications from dispatch to officers would have to be rerouted and on the fly adjustments made. Part of the reason for this success was frequent multi agency emergency drills which helped to refine emergency response and find weak areas to work on. Having regularly drilled together, the multi- agency network functioned more effectively during the emergencies that did arise. From my experience, the regular training utilizing the emergency procedures and equipment was a key factor in providing the effective response needed when it was no longer a drill, reduced the potential for human generated failures on the part of the emergency response teams and well worth the time and effort put into them.
What would you do if 911 failed? What steps would you take to prevent such a failure? I welcome your thoughts and comments below.