By Daniel Beekman
The Seattle Times
SEATTLE — Years ago, a technology officer inside Seattle’s government warned the city’s Fire Department that experimenting with artificial intelligence for 911 medical calls would pose equity risks.
She knew about research showing AI could perpetuate or exacerbate bias based on traits like race and gender, and she worried about that affecting life-or-death decisions. So in 2019, she recommended mitigation strategies, including that the department plan to share details with the public.
| READ NEXT: AI is an assistant, not an automator
Those recommendations apparently went almost nowhere.
Instead, the Fire Department later began using AI to help analyze and triage 911 medical calls without any disclosure to callers, any public review or any demonstration that the technology would treat diverse callers equitably.
Since 2023, AI has been listening to all 911 medical calls in Seattle and helping dispatchers decide which callers don’t require a rapid response.
The technology is prompting dispatchers to route certain 911 patients to a Texas -based nurse line rather than send them aid right away. Ambulances ordered by the nurse line are exempt from Seattle’s response-time standards; in one case, before the Fire Department started using AI prompts, a woman sent to the nurse line waited over 10 hours and was ultimately found dead.
Callers didn’t know about the Fire Department’s AI deployment until a Seattle Times report this June. At the time, Mayor Katie Wilson said her administration was developing a framework for assessing” how Seattle agencies are using AI. The City Council recently held a hearing on the topic.
The Times subsequently learned about the city’s years-old equity concerns through a request for public records.
Fire Department officials say their 911 system is safe because human dispatchers remain in control of all calls and are trained to handle them equitably.
But the technology officer who raised AI concerns in 2019 and several independent experts said they still have questions about the system. They say the Fire Department should stop using AI on live 911 calls until a thorough evaluation of risks is completed and shared with the public.
“This is something I and others have been worrying about for a long time,” said Emily Bender, a University of Washington professor who specializes in computational linguistics. “We know that speech technology does not work equally well for all language varieties.”
Wilson declined to be interviewed for this story.
“Mayor Wilson is highly concerned about the potential disparate impacts of AI algorithms on the public and is working with SFD on this,” Wilson spokesperson Ansel Herz wrote in an email last week, promising a public update in early 2027 on all uses of AI by Seattle’s government.
In general, Wilson believes that “thoughtful and moderated AI adoption” to serve residents better while protecting their privacy “is the best path forward.”
Watchdog’s 2019 concerns
Ginger Armbruster was Seattle’s chief privacy officer in 2019, working in the city’s Information Technology Department, when she became aware that the Fire Department was planning to deploy artificial intelligence.
A Danish company called Corti was building the department a new 911 call-taking platform, and the Fire Department wanted to use Corti’s AI to review calls for quality assurance. Armbruster was told it was a pilot project for completed calls, not live emergencies, she said in an interview last month.
Because it was a pilot and the city was still developing its approach to vetting new technologies, the project didn’t go through Seattle’s normal process for buying services and didn’t receive a comprehensive technology review, according to the Information Technology Department.
It received less scrutiny than it might have. When the city later considered using Corti software for police 911 calls, a surveillance technology determination report was posted publicly and council members were briefed.
Nonetheless, Armbruster did look at the Fire Department’s AI project in 2019. In a “privacy risk analysis” memo, she identified concerns related to data ownership, data sharing, data breaches and equity.
AI models would “present potential civil liberties concerns about equity of service provision and response dispatch,” Armbruster wrote. Their decisions, she wrote, could be affected by the use of language, accents and disabilities.
As mitigation options, Armbruster recommended that the Fire Department consider running its Corti project through a race and social justice assessment. She also recommended the department “work with public leaders to validate and socialize the use of this technology.”
Lastly, Armbruster recommended the department “prepare for media and public inquiries with documentation, case studies and examples of the benefits and uses of this technology in studies and live examples.”
Fire Department officials partly addressed her concerns in a 2019 data agreement with Corti. She said she never heard back about her equity concerns.
Although there was no hard requirement for the department to follow up, Armbruster believes she should have at least been consulted again in 2023, when the department started using AI prompts on in-progress 911 calls.
“The initial conversations were not anything to do with live data,” which would have intensified Armbruster’s concerns, she said last month.
Fire Department’s view
When asked about the 2019 memo, the Fire Department didn’t answer directly whether it pursued equity-risk mitigations related to its AI deployment. Instead, spokesperson Kristin Hanson outlined certain general practices.
The department’s dispatchers use call-taking protocols developed with oversight by the department’s medical director, Hanson said. The dispatchers, who are firefighters, remain the decision-makers on 911 medical calls, even when they receive AI prompts, Hanson added.
The dispatchers receive special training to answer 911 calls, including through a state certification program, Hanson said. They also follow policies from the state’s 911 coordination office, designed to help dispatchers serve callers with disabilities and who speak languages other than English.
The department “honors the spirit” of Seattle’s race and social justice initiative and has increased its efforts around reducing implicit bias and removing barriers to the department’s services, Hanson added.
Fire Department officials have cited research from Denmark that in 2023 showed Corti’s AI could detect stroke in emergency calls better than human dispatchers, across age and gender (the Seattle Fire Department’s medical director was a co-author of that study). They have not shared any research on Corti’s AI, nurse line triaging or equity, or on attributes such as ethnicity or accents.
UW Medicine is conducting an evaluation related to the Fire Department’s use of AI and its nurse line, Hanson noted. Neither the department nor UW Medicine have shared specifics or an estimated completion date.
“The public interest in how the 911 system in Seattle works is welcome. Public scrutiny, as uncomfortable as it can be at times, is part of how systems improve,” Hanson wrote in response to questions from The Times.
Fire Department officials recently asked the Information Technology Department to work on retroactive privacy reviews of its Corti platform, Hanson added.
Corti didn’t comment when contacted for this story.
What experts say
Bender, from UW , said her worry is a language model like Corti’s working less well for 911 callers whose speech patterns are different because of disabilities, accents or stress.
Shobita Parthasarathy, a professor who directs the University of Michigan’s science technology and public policy program, agreed with Bender. AI models have been shown to struggle to understand people with intellectual and developmental disabilities, Parthasarathy said.
In a 2022 study of Microsoft speech recognition’s performance with various speakers in Washington, UW researchers found much higher error rates for Black and Native speakers compared with white speakers.
Corti says its models are superior because they’re trained on medical conversations. Still, there’s evidence of Corti’s AI struggling with accents.
“Even if you have a (model) trained on a selection of speakers from Seattle who are calling 911, that’s still going to be more representative of some than others,” Bender said.
The danger of an AI model working less well for people with less typical speech patterns could be heightened in Seattle’s 911 system, where the technology is being used to help feed the Fire Department’s nurse line.
The nurse line is partly used for people who call 911 because they lack easy access to regular medical care, including homeless people and isolated older adults. They may be extra vulnerable and harder to understand, Bender said.
Especially in consequential scenarios like 911 calls, “You have to be very careful about whether the models are functioning in a way that does not discriminate” based on factors like how people speak or what neighborhood they’re calling from, said Tee Sannon, technology program policy director at the American Civil Liberties Union of Washington.
Although the Fire Department’s human dispatchers are in control, the automated prompts are “going to impact” their choices, Bender added.
Humans tend to assume machines are objective and smart, even when that may not be the case, said Bender, who coined the phrase “stochastic parrot” in 2021 to describe how large language models use statistics to mimic human speech without actually understanding what’s going on.
Research has shown “That when AI gives that prompt, the human is less likely to trust their independent judgment,” Parthasarathy said.
Bender worries the Fire Department’s use of the technology could also cloud questions about accountability when something goes wrong.
She’s glad Armbruster raised equity concerns in 2019, but she’s disturbed the city apparently didn’t do more to address those concerns, especially before starting to use automated prompts on live 911 calls in 2023.
“It gets even worse if this is being used in real time,” she said.
What should be done
To address equity risks, a best practice would be to collect demographic information from consenting participants, assess how a technology is working across groups and be transparent about the results, Bender said. It’s not easy to figure out an evaluation like that, but it’s not a mystery, she said.
If the Fire Department says it’s working fine,” Bender said, “it’s fair for ( Seattle residents) to say, ‘OK, then prove it.’ ”
Ideally, a public evaluation would have been carried out before the department fully deployed its AI technology, Bender and Parthasarathy said.
“The time to evaluate it was a while ago,” Parthasarathy said.
There should be a public audit, Sannon said. Seattle should suspend Corti’s live prompts until an assessment is done and residents’ trust is earned, Sannon, Armbruster, Bender and Parthasarathy said.
Armbruster thinks the Fire Department has deployed AI with the best intentions, hoping to make its 911 system more effective and efficient.
She believes the equity and transparency concerns she raised back in 2019 were simply overlooked as busy officials pushed ahead with their plans, processes changed and knowledgeable people came and went.
“That happens not infrequently with pilot projects,” said Armbruster, who said she lost her job this September when her chief privacy officer position was eliminated. “Things like this end up in no-man’s-land.”
© 2026 The Seattle Times.
Visit www.seattletimes.com.
Distributed by Tribune Content Agency, LLC.
