A Need for a Honest Look at How We Do Incident Management


Compared with other fields ours is still young and we haven’t figured out all the things just yet. The natural tight connection between academics, open source software and the improvements we’ve already seen can make it easy to think we’re already doing all of the hard work. All of which has been on specific technical challenges and very little on how we as an industry should improve how we work.

Consider the difference between how much attention we place on data we collect from the servers and services we support compared with what we have available for our entire field. We love dashboards and metrics to the point that they’re used to drive businesses and/or teams. Why haven’t we done the same thing at the macro level to help improve and guide our profession?

In 1895 a group was formed to look at standardizing the installation of automatic sprinklers in the United States. The results of these efforts was the creation of the National Fire Protection Agency and a set of rules for installing sprinklers that became what is now NFPA 13. This has been updated over the years, but it’s still the current standard for installing and maintaining automatic sprinklers.

Since then there has never been a multi fatality fire in a building that had a properly installed and maintained sprinkler system. The only exceptions are due to explosions and deaths from fire fighting.

The NFPA continued to learn from both large tragedies that created public outcry and other common fire conditions to shape the fire codes over the years. While I was a firefighter and fire inspector I was told that almost every part of the fire codes we used could be traced back to some event that prompted it to be written. Fires like the Triangle Shirtwaist fire (1911), Cocoanut Grove fire (1942), Our Lady of Angeles School fire (1958), Beverly Hills Supper Club fire (1977), Happyland Social Club fire (1990), and the Station Nightclub fire (2003).

As a response to the growing fire problem the country had through the 60s and 70s the National Commission on Fire Prevention and Control published a report called America Burning. It began with a set of recommendations on establishing a national organization and a data system to provide focus, review and analysis of the entire issue. From this recommendation the National Fire Incident Reporting System (NFIRS) was created and the Fire Service was then able to address larger issues that the data highlighted.

We have conferences, blog posts and books from our peers sharing their experiences on the issues they’re facing. This is wonderful and one of the reasons I’m drawn to the work that I do, but this does not address the need for an unbiased and professional response to the bigger issues we face. We could learn from both the example of the NFIRS and the CERT teams from our information security peers to create a group to study how we do incident management and provide recommendations.

Imagine if we had a way to voluntarily report incident data that could then be used for analysis. Although we’ve made improvements over years in how we discover, respond, and prevent incidents, we still have a long way to go in how we record our collective history and learn from it.