This article originally appeared in the Oct. 15, 2016 issue of the International Association of Fire Chiefs’ On Scene and is republished here with permission.
By Mark Chubb
Proving what we do works for our communities has always challenged fire and emergency services. Nevertheless, pressure to perform and prove that our performance affects outcomes for the better has never been greater.
The introduction of two new community risk reduction (CRR) standards, NFPA 1300 and the forthcoming NFPA 1730, has many fire marshals and fire chiefs asking how they can demonstrate the effectiveness of their programs. Executives in communities with populations of less than 50,000 have voiced particular concerns that the data required to drive interventions and prove their performance may be lacking.
Evidence-based interventions take many forms though. Regression-analysis and statistical tests of significance may be the gold standard of proof, but they don’t necessarily get you better results in the short-term. And we all know the short-term matters to most mayors, councils and city managers.
In the past, most fire service executives relied on anecdotes to illustrate worst-case scenarios. These usually played on emotions and signaled the downside risk of failing to accept tough policy choices. The problem with this approach wasn’t the anecdote; it was the way it was used.
Too often, in the effort to prove a point, the anecdotes relied on a strawman or hypothetical situation. If the case in point didn’t resonate with the intended audience, the analysis of what it meant was doomed. In contrast, the use of actual case studies that involve real people and real consequences almost always work better. When using case studies or anecdotes, the real questions are always, “What does this represent? How does this matter? And why should I care?”
Anecdotes that focus on the upside risk or benefit of interventions make the best case studies. Cases that illustrate how an intervention prevented something bad or worse from happening often resonate loudly and clearly with an audience.
It may be worth noting here that few mayors, councils or city managers have the time or attention to analyze or check the data they receive for precision or accuracy. In most cases, they think in much simpler numerical terms. A single occurrence is probably nothing but chance. Two occurrences could be nothing more than coincidence. However, three or more of something, especially something bad, usually looks like a trend.
Any case that can be presented as an archetype or example of a common or recurring situation has natural resonance with your audience. The more grounded in the present and familiar you make the facts or circumstances, the more likely it will be accepted as proof of your point.
As helpful as this advice may seem, you still have one big challenge to overcome in every evidence-based case: Proving that the addition or subtraction of one or more elements under your control will change the outcome.
Here’s where case studies make or break decisions. Multiple examples always work better. Two cases that vary only slightly in the presence or absence of a specific feature or intervention work better than any single case can. And any case that illustrates a string of data will always be considered stronger than an isolated case unsupported by other data.
Here’s a quick illustration of how this can work in practice:
As you know, the department responded to a two-fatality house fire in the early-morning hours of July 5. Responding firefighters found two adult males deceased in a second floor bedroom. Six other residents were removed from the dwelling, two of which suffered severe injuries due to smoke inhalation and burns. Medical tests and neighbors’ reports confirmed that all of the occupants were impaired by intoxicants consumed the night before while celebrating Independence Day with consumer fireworks.
Investigators probing the origin and cause of the fire believe errant fireworks ignited dried debris in the rain gutters, which allowed the fire to spread undetected into the attic where it was fueled by the plywood roof deck and timber roof framing.
Although the house was equipped with battery-operated smoke alarms in the bedrooms and hallways, these devices failed to provide early warning due to the location of the fire. By the time they would have activated, the ceiling collapse and fire spread from the attic overwhelmed escape routes.
This tragic incident was just one of 21 fireworks-related calls attended by the department from 2100 hours on July 4 when state law permitted discharge of legal consumer fireworks by the public. These incidents represent a 22% increase over last year and reflect a strong upward trend over the past three years in such incidents. Besides this fire, the department treated and transported three other minors who suffered burn injuries due to fireworks.
Although we increased staffing in anticipation of the higher call volume, automatic aid was required on three occasions during this period to handle simultaneous calls for service. Like us, our neighboring departments found themselves stretched thin, and response times suffered as a result. On average, it took units 15% longer than our 90th-percentile attendance time to reach calls due to the increased demand.
Six weeks ago, I testified before Council regarding public-education programs planned for this year’s fireworks season.
In the three weeks before Independence Day we launched a widespread media campaign, which included coverage by local television and print media. We posted placards and signs in prominent locations throughout the community, warning of the danger of fireworks, and we distributed safety leaflets at every stand selling fireworks.
Although state law permits the sale of certain consumer fireworks for a limited period of time each year, the statute also permits municipalities to enact local bans subsequent to a 12-month waiting period, during which time public comment must be sought. In light of recent experience, I strongly encourage Council to reconsider the decision not to enact such a ban and authorize the department to work with community groups to build support for such a policy change.
As presented, this case study references a specific incident and its consequences. It makes a clear reference to the cause and contributing factors. It addresses the question of mitigation. It places the incident in historical context relative to other incidents and the overall trend of incidents involving similar causes.
Once these facts are established, the case assesses the effectiveness of recent measures to address the risk and notes the unsuccessful outcome.
Finally, it offers a policy recommendation and clearly states a request for action.
Cases like this one always involve competing values. Some citizens will clearly oppose a policy change. Others will encourage and endorse it for reasons that have nothing to do with public safety. Nevertheless, in this presentation the fire department makes a clear and compelling case that fireworks pose a problem for citizens and responders alike and frames the decision for elected officials clearly.
Case studies and anecdotes can serve as a powerful call to action. And more importantly they give decision makers confidence in their decisions to reduce costs and consequences even in the absence of statistical tests of significance.
This article originally appeared in the Oct. 15, 2016 issue of the International Association of Fire Chiefs’ On Scene and is republished here with permission.
Mark Chubb, MPP, CFO, CFPS, FIFireE, FCMI, is a public-safety consultant based in Seattle and an at-large member of the Fire & Life Safety Section’s board. He’s been a member of the IAFC since 2010.
