Scientific American on the "broader problem" with AI errors: "Instead of checking AI's work, people keep trusting it."

[highplainsdem]

https://www.scientificamerican.com/article/why-lawyers-keep-citing-fake-cases-invented-by-ai/

May 22, 2026
AI keeps inventing fake cases. Lawyers keep citing them
The trend of attorneys getting caught citing AI-hallucinated cases points to a broader problem: instead of checking AI’s work, people keep trusting it

By Steven Melendez


-snip-

The pattern emerging across these cases is that people keep trusting AI’s answers even when they know the systems can be wrong. So far, that misplaced trust has led to dismissed legal appeals, attorney fines, fired journalists and software outages. Experts warn the stakes will rise as AI becomes more deeply embedded in professional work.

“Humans essentially have a tendency to believe that machines have more knowledge than they do, don’t break and are infallible,” says Alan Wagner, an associate professor of aerospace engineering at Pennsylvania State University.

-snip-

Research co-authored by Wagner suggests the problem could extend well beyond office work into life-or-death scenarios. In experiments inspired by drone warfare, his team asked participants to categorize images as civilians or enemy combatants and to choose whether to fire a missile at each potential target. A robot then provided feedback on each classification—feedback that was, in fact, random—and though participants’ initial assessments were mostly accurate, they reversed their views in most cases where the bot disagreed. The scenario was a simulation, but participants were “shown imagery of innocent civilians (including children), a UAV [uncrewed aerial vehicle] firing a missile, and devastation wreaked by a drone strike,” according to the paper. They seemed to take the task seriously, says study co-author Colin Holbrook.

“I think that’s the context in which those findings have to be interpreted,” says Holbrook, an associate professor of cognitive and information sciences at the University of California, Merced. “These people were really trying. These people thought that it mattered,” he adds. And if the scenario had been real, “they would have killed a lot of innocent people.”

-snip-