By Clint Smith and Ryan Falkestein-Smith
Artificial intelligence (AI) offers a wealth of opportunities to improve every phase of fire service operations when used as a trusted tool. However, if AI is not developed, deployed or maintained appropriately, there is a possibility of unintended consequences. To adopt AI as a trusted tool in the fire service, a proactive risk-management approach may help any unintended consequences by applying risk management.
AI risk management
AI risk management is the process of assessing the likelihood of an event occurring alongside the magnitude of its consequences. Consequences can be positive, negative or, in some instances, both, such as a “near miss” incident that could improve organizational learning at the expense of individual trauma for those involved.
AI risks are contextual, meaning that awareness of the environment and how AI is used is essential to framing the benefits and risks of the system being constructed. Perspectives on an AI system’s risk may differ across its lifecycle, from design and development through validation, deployment and operational use. Additionally, the difference in perspective may stem from the various participating individuals within the AI ecosystem, which include:
- The technology developer responsible for creating and training the AI models, delivering the functionality of the system
- The integrator responsible for manufacturing the final end-user product
- The end-user who uses the AI-enhanced tool
There is no unified perspective on what constitutes AI system risk. Because there is no clear definition of AI system risk, it is challenging to establish a single AI risk management process or procedure that addresses all risks. This is especially true for AI use in the fire service, specifically the end-user who encounters unique scenarios daily for which the AI system was not trained. The fire service needs an AI risk management process that’s adaptable to the particular application or mission for which the AI platform is being used.
Following a risk framework
One pathway to managing risks of AI systems is to leverage the NIST AI Risk Management Framework (AI RMF), which was created at the direction of the National Artificial Intelligence Initiative Act of 2020. The Framework was designed to equip organizations and individuals with approaches that increase the trustworthiness of AI systems, and to help foster the responsible design, development, deployment and use of AI systems over time.
This general-purpose framework consists of four functions:
- Map: Identify and understand the specific context in which the AI system will operate and the risks inherent to that environment.
- Measure: Assess the applications of methods and metrics used to analyze and monitor identified risks.
- Manage: Prioritize and act upon the assessment to respond to risks when they arise.
- Govern: Establish the policies, processes and procedures across the organization to manage AI systems effectively.
The core functions of the AI RMF can be adopted when using AI systems in the fire service, with these considerations in mind.
Mapping: In the fire service, defining the level of autonomy of an AI system provides insight into mapping the risk posed by that system. The level of scrutiny applied to a system’s application may be directly proportional to its autonomy.
Drawing from the Food and Drug Administration medical device classifications and the Nuclear Regulatory Commission’s assessment of autonomy levels, the fire service could categorize AI tools into distinct classes using the table below:
Measuring: In addition to the calculated metrics of an AI system’s accuracy, specificity and sensitivity, the consequences of failure and the various ways to assess them are an approach to measuring the risks of an AI system. An AI system can only work as well as it is designed and trained to work, suggesting that a poorly constructed model may fail. In the event of incorrect output or incorrect actions by the tool or by a user relying on an AI system, severe consequences may result. Based on the application’s context, the risk measurement process may vary. Risks associated with life-safety-critical tasks may warrant stricter evaluation metrics than those for low-priority functions. In either case, however, the consequences of the AI system’s failure may still apply.
Managing: Prioritizing and actively mitigating risks identified during the mapping and measurement functions creates a pathway for managing risk with AI systems. In the context of an AI system’s use in the fire service, the life safety classification may establish its risk assessment priority. A suggested ranking of the relative importance of each task or function could be categorized as follows:
- High: Life safety tasks (critical or immediately dangerous to life or health)
- Medium: Very important but not life safety
- Low: Beneficial but is generally not more important than other tasks
High classification categories may require greater scrutiny to avoid severe consequences in the event of failure. The responsibility for managing risk isn’t necessarily on users. Still, it may require collaboration among technology developers, integrators and end-users to work as a unified ecosystem to identify strategies that actively mitigate risks at every priority level.
Governing: Governance of AI systems to manage their risks ensures that other listed core functions of the AI RMF are applied consistently throughout a system’s use. Establishing a proactive plan to rigorously test, understand and properly validate AI systems before deployment is critical to their adoption. Similar to the management of core functions, the success of ensuring proper procedures are in place when implementing AI systems may depend heavily on a unified ecosystem sustained by strong communication among all parties involved in the AI system’s life cycle.
In summary
The integration of AI into the fire service as a tool offers a transformative opportunity to enhance situational awareness and firefighter safety. The success of its adoption, however, is rooted in a system’s recognized value, which may depend on a proactive risk-management approach.
By leveraging the NIST AI Risk Management Framework’s core functions, the critical work of managing risks may be achieved by:
- Mapping systems at different levels of scrutiny
- Measuring the consequences of potential failures
- Managing risks through clear prioritization of tasks
- Governing these systems through a continuous collaboration amongst the participating actors, who all consider proactive approaches to evaluating the identified risks and various levels throughout the AI system’s lifecycle
By committing to these core functions, the fire service has a pathway toward AI becoming a force-multiplier in a deep toolkit of resources.
ABOUT THE AUTHORS
Dr. Ryan Falkenstein-Smith is a mechanical engineer in the Firefighting Technology Group of the Fire Research Division (FRD) at the National Institute of Standards and Technology (NIST). As the leader of the Smart Fire Fighting Project, Dr. Falkenstein-Smith focuses his research on developing measurement science designed to enhance situational awareness, operational effectiveness and firefighter safety. Having extensive publications in sensor technology, Dr. Falkenstein-Smith has conducted research that enables the fusion of data from various sources with reliable predictive artificial intelligence (AI) models. Dr. Falkenstein-Smith is also an active member of the NFPA’s Electronic Safety Equipment Committee and AI Task Group and the IAFC Technology Council AI sub-committee.
Clint Smith, PE, is the CTO for NextGConnect, focusing on Artificial Intelligence (AI) and Edge Computing with resource constrained devices. He is a licensed professional engineer in New York and New Jersey. He has earned more than 175 patents and is also the author of nine wireless engineering books published by McGraw-Hill. Smith is a member of the NFPA’s Electronic Safety Equipment Committee and AI Task Group. He is an active volunteer firefighter with the Pine Island Fire Department.
