By Division Chief (ret.) Jim Brown, Dr. Ryan Falkenstein-Smith, David Little and Battalion Chief Chad Crouse
The modern fireground is no longer just a battle of heat and hydraulics; it’s also rapidly becoming a data-rich landscape fueled by artificial intelligence (AI). This creates a new training challenge: How do departments define what personnel need to know and be able to do with the technology?
| READ MORE: AI success starts with permission from the top
One approach is to identify the knowledge, skills and attitudes (KSAs) personnel need at different levels of AI competency, then translate those competencies into job performance requirements (JPRs). Those levels should be task-dependent rather than rank-dependent, meaning any member may need foundational, operational, technical or strategic AI capabilities depending on the work being performed.
A fire service AI training framework could mirror NFPA 1072’s progression from awareness through more specialized capabilities:
- An awareness-level member might demonstrate knowledge of AI system capabilities and limitations.
- An operational user might demonstrate the ability to use AI to analyze standard operating guidelines and identify needed revisions.
- A technical specialist might demonstrate the ability to build a secure, repeatable workflow.
- A strategist might demonstrate the ability to evaluate an AI use case for mission fit, risk, bias and governance.
Awareness-level user
Foundational literacy: Recognition, not application. This is needed for all personnel.
Knowledge: Fire service members may understand the basic qualities of today’s AI solutions, including that large language models are predictive systems that excel at language and pattern recognition. While AI may appear to be “thinking,” it simply generates responses based on patterns in its training data. Members may know the strengths of AI (e.g., summarization, drafting, pattern recognition) as well as the limitations (e.g., hallucinations, missing context, overconfidence). Critically, awareness-level members should recognize AI’s tendency to confidently fill gaps, producing plausible-sounding but incorrect answers without flagging uncertainty. This tendency is baked into how the AI systems work, and recognizing it is the first line of defense against being misled.
Skills: At the awareness level, the emphasis is on recognition, not application. Members are not responsible for using AI operationally. They may be able to identify when AI is being used around them and recognize outputs that may be unreliable so that they can appropriately flag concerns. Awareness training can extend beyond the fire department to the whole of local government
Attitudes: AI’s behavioral tendency toward confidence, answering even when there is uncertainty, requires a specific human counterweight: healthy skepticism. Awareness-level members can approach AI output the way they approach any unverified report: with interest but not automatic trust. Curiosity about what AI can do is appropriate at this level; deference to what it produces is not.
JPRs:
- Define AI terminology and core concepts.
- Recognize hallucinations and bias in AI output.
- Identify public records implications of AI use.
- Understand consent considerations related to data input.
- Recognize sensitive data and cybersecurity categories.
- Identify agency AI policy and acceptable use boundaries.
Operations-level user
Applied AI use: Structured prompting, validation, policy-compliant documentation. This may be appropriate for personnel with administrative functions.
Knowledge: Operational users may consider when AI output should be verified, especially when it touches policy, legal exposure, records, operational recommendations or data-informed conclusions. They may know when AI is an appropriate assistant and when it is not. Understanding AI knowledge limitations — that the system does not know your department, your jurisdiction, your SOPs or your operational context unless you provide it — is essential at this level.
Skills: Personnel could be able to write a clear prompt, provide context, state the goal, define the audience and improve results through iteration. Structured prompting is not just about getting better answers; it is about constraining AI’s tendency to comply with whatever framing the user provides. A vague prompt invites the system’s tendency to make assumptions and fill gaps. Practical skills include summarizing documents, extracting key points, organizing information, supporting research, and converting raw information into usable products. For example, an operational user may be able to upload a standard operating procedure, ask AI to identify gaps relative to a referenced standard, review the output and document what was used and what was modified before the product is finalized. Personnel may also be able to verify outputs and edit them for factual accuracy, tone and organizational relevance, treating AI output as a draft that requires human review, not a product that is ready to use.
Attitudes: AI’s tendency toward compliance — producing output that matches the user’s framing rather than challenging flawed assumptions — requires a specific human counterweight: accountability. Operational users can maintain ownership of the work product. The fact that AI drafted something does not reduce the user’s professional responsibility for its accuracy. At this level, the attitude of “I verified this” should never be replaced by “AI said this.”
JPRs:
- Construct structured prompts with context, goal and audience specified.
- Validate AI outputs for factual accuracy and policy compliance.
- Detect inaccuracies or hallucinations before use.
- Apply data screening procedures prior to input.
- Document AI-assisted work products per agency policy.
For example, someone planning a community event may use AI to conduct research to assist with scheduling, researching the topic and preparing the presentation.
Specialist-level user
Implementation and system design: workflows, agents, integration, validation. This may be appropriate for data analysts, accreditation managers, IT specialists and AI program managers.
Knowledge: Technical specialists should understand the governance, privacy, security, procurement, data quality, implementation planning, use-case selection, evaluation methods and human oversight requirements. They may not just understand what AI does, but also how AI systems affect the organization, including system breakdowns and organizational failures, data exposure risks and the ways AI behavioral tendencies can scale into institutional problems when not maintained. Specialists may understand how AI’s skills can be combined into workflows that go beyond individual prompts, including retrieval-augmented generation, agent architectures and multi-step automated processes.
Skills: The technical specialist skill set may include building structured workflows, managing data preparation, creating repeatable prompts, configuring retrieval systems, supporting dashboarding and developing custom tools and agents. These are similar to operations-level skills but require a deeper level of knowledge for specialists. Specialists might be able to design a validation process for AI outputs, test systems against known inputs and outputs, identify where human review is required before action and document system behavior well enough to support training and audit.
Attitudes: AI’s confidence in its output indicates the need for a specific human counterweight at the specialist level: systematic oversight. Specialists who build systems around AI may consider designing for human oversight (i.e., human-in-the-loop) rather than assuming the system’s confident outputs are reliably correct. Well-designed systems anticipate failure points and have rollback plans.
JPRs:
- Design AI workflows and agents for specific fire service use cases
- Integrate AI with databases and records systems
- Conduct structured risk assessments for AI implementations
- Implement cybersecurity controls appropriate to data sensitivity
- Test and validate AI systems against defined performance standards
- Establish documentation standards for AI-assisted processes
Strategist-level user
Enterprise governance: Roadmaps, readiness, risk, workforce and sustainability. This may be appropriate for personnel with management responsibilities.
Knowledge: The AI strategist shares the specialist’s foundation in governance, privacy, security, procurement and human oversight, but the focus is on integration with long-term organizational plans. Strategists may have a vision not just for how individual AI systems work, but how AI adoption at scale interacts with workforce development, institutional culture, public accountability and mission continuity. Strategists may consider that AI behavioral tendencies — helpfulness, compliance, gap-filling and overconfidence — do not stay contained to individual interactions. When AI is embedded in institutional workflows without adequate oversight, those tendencies scale. An agency that relies on AI for policy drafting without verification processes is not just accepting individual errors; it is building institutional blind spots.
Skills: At the strategist level, skill is about aligning AI use with mission needs. Strategic AI skills include prompt science, selecting appropriate use cases, setting boundaries, analysis, assigning oversight, defining acceptable risk, evaluating vendor claims and building implementation plans that connect training, policy and accountability. Strategists should be able to evaluate an AI use case for mission fit, risk and governance before approval, asking not just “can AI do this?” but “should we use AI for this, and what oversight structure ensures a corrective failsafe when an initial failure occurs?”
Attitudes: AI’s tendency to mirror user assumptions, producing outputs that validate the framing it is given rather than challenging it, is the most potentially consequential tendency at the strategic level because strategic decisions carry the most organizational weight. Strategists can model the attitude of institutional skepticism: build processes that assume AI will sometimes be wrong in ways that are difficult to detect, and design governance accordingly. Service orientation also ensures that AI strengthens the mission rather than optimizing for efficiency at the expense of accountability, providing a human counterweight to AI overuse.
JPRs:
- Develop AI implementation roadmaps aligned with agency mission, values and community expectations.
- Conduct organizational readiness assessments prior to AI deployment.
- Design governance and oversight structures that account for AI behavioral tendencies.
- Evaluate legal and reputational risks of proposed AI use cases.
- Conduct return on investment and sustainability analysis for AI investments.
- Anticipate workforce and second-order impacts of AI adoption.
Applying the framework in a use case
Using an AI-assisted note-taker as a use case example, the KSAs at each level could be as follows:
Awareness (all personnel): At this level, employees need to understand the risks and benefits associated with AI note-taking systems. They should understand that while an AI note-taker can effectively capture minutes for shift briefings or training debriefs, it also captures everything being said aloud. Participants using the tool should be aware of the data security risks associated with sensitive information. Furthermore, all participants should know their rights when requesting that a note-taker be stopped.
Operations (any person at any rank who uses the tools): At the operations level, the employee needs to know their responsibility in managing the note-taker. The person who owns or initiates the note-taker is ultimately responsible for understanding the department policy regarding their use. The operator who implements an AI note-taker in a meeting should be responsible for announcing its presence to all attendees.
Specialist (a person or team with domain-specific KSAs): The specialist is responsible for researching AI note-taking tools, their features and data retention policies. The specialist understands the secondary effects of different AI systems. For example, the specialist may manage the AI note-taker’s output (e.g., structured notes, assignments, audio recordings) and define the file-naming convention to ensure consistent record management.
Strategist (a person or team with domain-specific KSAs for governance and mission alignment): At the strategist level, the employee or team is responsible for understanding the department’s strategic plan and priorities. They must understand the role of governance and are responsible for creating the policies and governance to support the AI note-taker in concert with their department’s mission and values.
The incorporation of JPRs into training transforms AI technology adoption from a novelty into a disciplined professional standard. By structuring AI competency into task-dependent levels, the fire service can ensure that every member, regardless of rank, possesses the tailored knowledge, skills and attitudes required for their role. Ultimately, JPRs provide a sustainable pathway for building institutional trust and mission-focused accountability. This framework ensures that AI use is not arbitrary and that research, testing and training procedures appropriately guide its adoption and use. As with any tool used in the fire service, successful adoption relies on training and education. Why should AI be any different?
ABOUT THE AUTHORS
Jim Brown is a retired division chief from Monterey, California, now living in Hawaii. He is the lead for the Fire Service Working Group of the GovAI Coalition and a member of the IAFC Technology Council’s Data and AI subcommittee. Brown is a California-certified chief officer and master instructor, holds a bachelor’s degree in fire science from Columbia Southern University, and is a contract instructor for Analytical Tools for Decision-Making at the National Fire Academy.
Ryan Falkenstein-Smith, PhD, is a mechanical engineer in the Firefighting Technology Group of the Fire Research Division (FRD) at the National Institute of Standards and Technology (NIST). As the leader of the Smart Fire Fighting Project, Dr. Falkenstein-Smith focuses his research on developing measurement science designed to enhance situational awareness, operational effectiveness and firefighter safety. Having extensive publications in sensor technology, Dr. Falkenstein-Smith has conducted research that enables the fusion of data from various sources with reliable predictive artificial intelligence (AI) models.
David Little is a retired CEO at ISG/Infrasys Thermal Imaging, one of the first companies globally to provide thermal imaging to the fire services, sold to Scott Safety in 2014. Little has 35 years of experience in the fire service, 30 focused on thermal imaging. He currently serves as chair of the AI Task Group, NFPA Electronic Safety Equipment Technical Committee.
Chad Crouse is an operational battalion chief with the St. Lucie County Fire District in Florida and chair of the IAFC Technology Council’s AI and Data Subcommittee. A former division chief who oversaw IT, communications, community risk reduction, and emergency management, Crouse pairs administrative experience with daily field command — a perspective he brings to his writing and national presentations on data and AI. He is also a contract instructor for the National Fire Academy’s Managing Officer and Executive Fire Officer programs.
