The recruit academy you are assisting with is going well. Students are attentive and working hard, and the schedule that you and the other members of the cadre put together seems to be working.
You start the recruits working through some hands-on firefighter survival scenarios when things start to fall apart. Some students are floundering while others are displaying bad habits that didn’t show up before, even when they perform simple tasks outside the stressful environment. What’s going on? Was it your instruction? Something unknown about the students? It might be that things were not going as well as you thought because you were not properly evaluating the learning.
If that sounds familiar, you are not alone. In this year’s What Firefighters Want survey, which focused on training, 21% of respondents said their department has no consistent method for evaluating performance during training.
Teaching isn’t the same as learning
The goal of training is, of course, learning. However, measuring this learning has to be done carefully, and it’s important to make sure we aren’t just teaching, but actually allowing people to learn.
Teaching in the fire service is unique, for sure, in the topics and methods used to teach skills and knowledge. How the students learn, however, is not unique. Adult brains learn differently from kids, and the methods and motivation for learning can also differ.
When we are deploying a new tactic or training on concepts in an academy environment, one of the most important metrics is whether students are ready to move on to the next concept or need more time on the current topic. This is important because our progression to new knowledge or skills often builds on the foundation of previously learned knowledge and skills. Go too fast, and some will not have the solid base to effectively learn and integrate new skills. This creates a stressful environment and will affect students’ ability to learn.
The risk of adding stress too soon
We know that we operate in a stressful environment during an actual incident. This requires that, at some point, we add stress to training evolutions so that we can learn to apply skills correctly in an actual incident. However, if we add stress too soon, before the skill is integrated, we can create opportunities for mistakes and bad habits to become ingrained. Worse, we can create anxiety around a survival skill that makes it likely an individual may not perform that skill correctly when faced with a real situation. Too often, this happens when we try to move forward and stick to a schedule before we ensure that all students have learned the skill and are comfortable with it.
To measure this, we can wait for an emergency scene and hope the situation presents itself, or we can design evaluations. The most efficient way is to design a system that uses evaluations to determine when learning has actually occurred, then assess knowledge and skills over time.
Why evaluate?
Knowing when we have achieved learning, rather than just completed teaching, determines when we can move on to the next topic and when a hands-on skill is ready to be tried in a more stressful environment. It can also measure how effective our instructors are. Look at it from the standpoint of continuous improvement. If we evaluate students and instructors, the results can give us data to improve over time and make our training more effective.
Evaluation also creates accountability. I have been part of a number of academies where it was made clear from the outset that reading the textbooks was important and would affect test scores. Not all the students would take that to heart, while others would do their best to stay on schedule by reading chapters. When the first written evaluation came, those who didn’t read struggled, and this made it clear what they needed to do to be successful. It created accountability far more effectively than a cadre member harping on them to read their book.
On the flip side, it can also create a sense of accomplishment, camaraderie and esprit de corps when a group of recruits is tested on a difficult, hands-on skill and passes. The evaluation will give them confidence that they met a standard as a group.
How often do you train, bro?
When we learn a new skill or knowledge point, reviewing it 24-28 hours after the initial training can increase retention from 30% to over 60%. Could you build this into an academy schedule, reaching back to previous days to make sure you revisit some skills after a day or two to further cement those skills?
Our skills and knowledge are never permanent. We learn and forget information all the time. The more stressful or complex the skills, the more frequently they need to be refreshed. These are called perishable skills. We will lose proficiency or develop poor habits and shortcuts if we do not review them regularly.
However, it can be hard to know what this interval is. If we use written or observational evaluations, we can determine the average time before the skills begin to degrade, which can help decide how frequently that training should occur. This can replace the typical plan of training on some skills annually with a more accurate interval. This could free up space in a training calendar for different topics.
Army Special Operations Command, for one, frequently refreshes its high-risk/low-frequency skills quarterly. How often should we train on skills like SCBA confidence or window bailouts? If we use a quick, anonymous evaluation while observing the evolution of one of these skills, the aggregate data can help determine how quickly that skill begins to degrade.
This type of evaluation can be an important driver of a very effective training plan for the year. Skills can be practiced when needed to keep crews sharp, while less perishable skills can be drilled less frequently.
How do we evaluate?
When we discuss evaluations, many people feel an emotional response. From performance evaluations to thoughts of test anxiety or maybe even fear of performing in front of others, there can be a near-universal pushback to the idea.
Evaluations are critical for several reasons. One is to ensure that we progress at the right pace. Too often, because of limited training time, we stop teaching or practicing a skill once everyone has performed it correctly. This is, for many, a minimum level of competence. Training on a skill until we can’t make mistakes is another level of competence.
If we add stress to an evolution or try to perform a skill in less-than-ideal conditions, this can cause performance to break down and, worse, lead students to develop a bad habit, such as removing gloves or facepieces. This means that the firefighter now has a neural pathway built for performing a bad habit. How can we avoid this? Even simple evaluations can tell us when a group of students has reached a high level of proficiency and is ready for the evolution to be ramped up to a more realistic level to cement those skills in a real-world environment.
Evaluations don’t have to be stressful, miserable affairs. A simple survey or a check sheet that can be filled out from a distance might be all we need to measure if we are creating learning, and if that learning is still sticking.
Formal vs. informal evaluations
It’s important to understand what you are trying to accomplish with an evaluation and which tool you need to use to achieve that goal. Evaluations can be formal or informal. Formal evaluations are frequently the type associated with anxiety and are probably already in place in your organization. Things like probationary performance reports, annual performance evaluations, and quality improvement for response and medical care are examples.
In training, we will have times when we must use formal evaluations. Clearing someone to perform a new patient care skill or determining when they are qualified as an apparatus driver/operator are examples. We need a method to determine if skills meet a minimum threshold before we consider someone qualified. This helps to ensure confidence and trust.
Make sure you understand the rules that apply to these types of evaluation in your agency. State record-keeping requirements, as well as collective bargaining agreements, can all apply to this type of evaluation.
Subjective vs. objective
We don’t always have to use formal evaluations. Like any tool in the toolbox, they are not always the best choice for every situation.
Evaluations fall into two categories: objective and subjective. They both have their place, but it’s important to understand the difference between the two.
An objective evaluation relies on measurable factors. Things like test scores, ratings on a 1-5 scale, or an elapsed time for an evolution or skill are great examples. The testing method should rely less on judgment and more on repeatable, measurable factors. This means that objective test results can often be used to compare things and measure improvement.
An example could be a recruit evaluation spreadsheet that tracks written test scores and hands-on skill performance over several weeks at an academy. Looking from week to week for an individual student will show trends of improvement or whether a recruit is struggling. This can be a great basis for a conversation with the recruit, to see if there is a reason they are struggling, or to commend great performance.
Reviewing each week’s results makes it clear which recruits struggled or excelled. This can identify material or skills that need more work, or an instructor who may not be connecting with the students.
Subjective evaluations are more dependent on a personal impression, or items that can’t be easily measured. This means they can be less replicable but are still an important tool for evaluating the intangibles of a student or instructor. Things like team performance and attitude could be measured with a subjective tool. However, subjective evaluation can easily be swayed by biases and unconscious factors. Things like race or even familiarity with some students vs. others can cloud results. Creating a set scale can help weed out these issues. A simple 1-5 scale for team performance could look like this:
- Frequently has conflicts with peers or has difficulty working with others.
- Rarely has conflicts, and sometimes works effectively with other team members.
- Does not have conflicts and can cooperate with others on tasks.
- Has no conflicts, and displays initiative and can lead some cooperation on tasks.
- Has no conflicts, and consistently displays initiative and high-level cooperation with the team.
It’s not perfect, but having simple description like this can help an evaluator be consistent in their evaluations.
When we are evaluating a hands-on skill or evolution, remember that it’s important to have a standard to evaluate against. Firefighter I and II job performance requirements, or NFPA 1410 evolution times, can be a great starting point. Just make sure there is a set standard.
Survey says
If we were honest, we would probably say we don’t prefer being evaluated, but the results of the What Firefighters Want survey seem to suggest otherwise. Respondents were asked: “What one change would most improve the quality of instruction in your department’s training program?”
A number of the responses discussed evaluations, with many people asking for more evaluations:
- “Give operational skill exams and go over skills again and again.”
- “Require a formal performance evaluation.”
- “Improved constructive feedback from instructors.”
- “Holding individuals and crew accountable for when their evolution is below acceptable. These folks may be talked to, but no actual accountability and course of corrective action is ever taken.”
- “Instructor evaluations and recertification.”
- “Policies and SOPs regarding how training is conducted, how it is evaluated, and how the training schedule is decided for the year.”
- “Written evaluations that hold people accountable.”
- “Have a vision of what a successful completion looks like, backed by policy and research. Often, ‘success’ is based on the opinions of those involved with evolution, which changes with every evolution.”
- “Objectives set forth ahead of time with specifics on what we expect to learn.”
- “Immediate feedback and FOLLOW-UP TRAINING ON WEAKNESSES.”
The survey also asked how performance is evaluated during training. Of the more than 1,300 respondents who answered, 43% said informal observation and feedback best describes it, and 36% said objective skill-based evaluations. Another 21% said there is no consistent evaluation method, and fewer than 1% said written tests only.
Evaluate to know the learning stuck
Training is not finished when the material has been taught; it is finished when the learning has been shown to stick. Evaluation tells you when recruits are ready to move on or add stress, whether your instructors and material are connecting, and how quickly perishable skills begin to degrade, so refresher training can follow what crews actually need instead of a default annual cycle. Objective measures such as scores, ratings and times show trends over time, and a defined scale keeps subjective judgments consistent. None of it has to be stressful or formal; a short survey or check sheet is enough to start. Pick a skill, set the standard it will be measured against, evaluate it and then evaluate it again later to see whether the learning held. Many of the firefighters surveyed asked for exactly that: objectives set ahead of time, constructive feedback and follow-up training on weaknesses.
