MAHA Report Had Made-Up Citations
It is as if qualifications for positions of power matter.
NOTUS reports: The MAHA Report Cites Studies That Don’t Exist.
Health Secretary Robert F. Kennedy Jr. says his “Make America Healthy Again” Commission report harnesses “gold-standard” science, citing more than 500 studies and other sources to back up its claims. Those citations, though, are rife with errors, from broken links to misstated conclusions.
Seven of the cited sources don’t appear to exist at all.
That’s not the “gold-standard” of anything. This is the kind of thing that would lead an undergraduate to get a failing grade, and certainly this is not what one expects from the highest level of government. Although I will say that the official response does kind of sound like something a student might try to pull.
White House press secretary Karoline Leavitt and Health and Human Services spokesperson Emily Hilliard later acknowledged some of the citation inconsistencies but dismissed them as formatting issues.
It is well known that formatting errors often result in studies being made up out of whole cloth!
All of this is yet again a reminder of why competence matters.
There is a cavalcade of errors recounted in the piece.
Epidemiologist Katherine Keyes is listed in the MAHA report as the first author of a study on anxiety in adolescents. When NOTUS reached out to her this week, she was surprised to hear of the citation. She does study mental health and substance use, she said. But she didn’t write the paper listed.
“The paper cited is not a real paper that I or my colleagues were involved with,” Keyes told NOTUS via email. “We’ve certainly done research on this topic, but did not publish a paper in JAMA Pediatrics on this topic with that co-author group, or with that title.”
[…]
The anxiety study wasn’t the only one the report cites that appears to be mysteriously absent from the scientific literature. A section describing the “corporate capture of media” highlights two studies that it says are “broadly illustrative” of how a rise in direct-to-consumer drug advertisements has led to more prescriptions being written for ADHD medications and antidepressants for kids.
The catch? Neither of those studies is anywhere to be found.
Quite honestly, my first assumption is that someone used an AI tool. They are known to make up citations.
It’s not clear that anyone wrote the study cited in the MAHA report. The citation refers to a study titled, “Changes in mental health and substance abuse among US adolescents during the COVID-19 pandemic,” along with a nonfunctional link to the study’s digital object identifier. While the citation claims that the study appeared in the 12th issue of the 176th edition of the journal JAMA Pediatrics, that issue didn’t include a study with that title.
As the Trump administration cuts research funding for federal health agencies and academic institutions and rejects the scientific consensus on issues like vaccines and gender-affirming care, the issues with its much-heralded MAHA report could indicate lessening concern for scientific accuracy at the highest levels of the federal government.
That last clause may be the understatement of the year!
NOTUS also found serious issues with how the report interpreted some of the existing studies it cites.
In one section about mental health medication, which Kennedy has railed against for years, the report cites a review paper it claims shows that therapy alone is as or more effective than psychiatric medicine. But one of that paper’s statisticians told NOTUS that conclusion doesn’t make sense, given their study didn’t even attempt to measure or compare therapy’s effectiveness as a mental health treatment.
“We did not include psychotherapy in our review. We only compared the effectiveness of (new generation) antidepressants against each other, and against placebo,” Joanne McKenzie, a biostatistics professor at an Australian university, said in an email.
There is a lot more like that in the NOTUS piece.
Nevertheless,
Kennedy has enthusiastically promoted the report, calling it a “milestone” in a post on X after its release.
“Never in American history has the federal government taken a position on public health like this,” Kennedy wrote.
That may well be true, but not in the way Kennedy intends.
When you begin an “investigation” with a conclusion, then go look for supporting “evidence,” odds are you’ll have to make up some of it.
Almost certainly an AI tool. If a student submitted this, they would get a failing grade. If a professional submitted this to a journal, they would be drummed out of the profession, or at least shunned (in the case of tenure).
I’m uncertain that this would be enough to fire a tenured prof, though.
“They said there would be no fact checking!” – the motto of this crew.
More flood the zone with bull poo poo. Watch the sycophants begin pointing to this report to justify lots of bad policy choices. 1984 is 21 years late, but it has arrived in spades.
But we’ll soon be getting that autism cure! I can’t wait to suddenly acquire the ability to file my taxes. Goodbye TurboTax!
@Sleeping Dog:
Well…..
This is why I want the death penalty for self-proclaimed scientists who manufacture data or do crap like this. And no, I’m not joking. This is Lysenkoism in action.
Sounds like whomever did it had the same qualifications of a Harvard president. You know, the one they didn’t remove from the college but just busted her back to professor after her plagiarism was exposed.
@Steven L. Taylor:
Math was never my strong suit…
It’s almost as if religious nuts, conspiracy nuts, Third Reich enthusiasts, grifters and the morons they fleece can’t do science. I for one am shocked.
@JKB:
What do we bust Worm Brain back to?
I’m waiting for some MAGA to proclaim that Dr. Katherine Keyes is a liar trying to sabotage Trump.
She’s just a minion of the Deep State, you know.
Given that all the major journals are apparently ridden with corruption/tools of pharma etc, to the point that he plans to stop NIH researchers from publishing in them, it’s surprising he used those same journals for this ‘report’.
Of course, quoting nonexistent papers is such a classic failing of AI I don’t know how anyone could see the report as anything other than AI generated.
Generative AI tools absolutely just make up citations to nonexistent works, and cite works that do not actually support the argument being made. It is wild to me that the tool creators are seemingly totally okay with it.
@Steven L. Taylor:
21 is inclusive in 41.
Besides, my time sense works like that. I swear the 90s were like ten years ago.
My personal favorite:
It’s a miracle!
I mean, who would ever find out that they were making this sh*t up?
No wonder these low-life HATE fact-checking.
@JKB: Not even close. Harvard prof didnt properly cite some stuff and used some stuff out of Wikipedia, the same as Altman’s wife, who is still a professor. In this case entire citations were made up and false claims were made up about what many cited articles claimed. So neither of the Harvard profs, the president or Altman’s wife actually made up stuff or lied about the contents of a cited article. Also, a good number of the articles cited are sketchy at best.
Steve
I don’t use generative AI, but can’t you just instruct the AI to not hallucinate? (Meaning, are these people so clueless they aren’t even using the AI properly?)
This is, of course, no surprise at all. The tech bros who are now running amok in DC have way oversold AI. (Just go back to Speaker Johnson saying Musk was welcome to use AI to root out fraud across sensitive data fields.)
All of this is completely unacceptable behavior. In a just world, everyone involved would have both apologized and been fired by now.
I regret to inform you that this clearly shows that competence doesn’t matter.
@Jen:
If it was this simple, I would expect that someone would have introduced the innovation of silently adding that “and don’t just make shit up, use real information” to every query.
@Gromitt Gunn:
They’ve invested too much money to not get a return on that investment.
Sure, it’s easier to sell a product that isn’t fundamentally broken, but that’s just not an option, so they will sell what they have.
(Relatedly, I have some concerns about self-driving cars, but I think there’s a decent chance that they can get to the point where they make fewer (but different) errors than human drivers, and respond faster and more consistently to other drivers doing stupid things.)
@JKB: Is whataboutism all ya got?
@Gustopher: point taken.
Should matter.
@JKB:
Come on man, this is rookie level, Fortune level. Dig deep, man, you’re the boss dawg troll in these parts. I got faith in you.
@Jen:
You know, I never tried that.
No idea whether it would work. Probably not directly, if it doesn’t even know it’s hallucinating (and assuming it even knows anything).
What might work is to request links to all papers, cases, works, etc. cited. Depending on the bot, it may already do this. Copilot provides links to some of the statements it makes, for instance. But then one has to click on all links and check that they lead to the correct document.
Further, one needs to read said documents and determine what they say and what position, if any, they support. And then one needs to edit the report accordingly. And if one has to think and judge and write, then what need for generative AI?
So, seriously, this part is particularly upsetting for me. I worked with a therapist for years. One of the interesting bits of that relationship was how as we did all that hard work it became clear to her, and then me, that I have ADHD and always had it. It was just masked by abuse, PTSD and dysphoria. Unpacking and dealing with those things helped show the biological problem underneath.
Getting on proper meds showed me how neurotypical people function. Having had that experience, now I’m without meds, I’d rather not live. This is absolute hell. It’s deeply insulting.
And by “milestone” he meant “millstone.”
Who could’ve guessed we’d get whataboutism in this thread that is so remotely incomparable that I didn’t recognize it as an analogy. Yes, I’ve heard of people lying on résumés and cheating on papers. No, an official government document used to radically fuck up Healthcare in this country shouldn’t have the same ethical issues.
@Jen:
AI = Automated Idioacy
It is such an an oversold bubble in my mind. Yes, it is useful and has great capabilities, but this underscores it’s weaknesses….and/or its dangers, when willy nilly trusted to provide factual and proven answers.
@Kathy:
@Gustopher:
I knew I’d seen something about tailoring prompts, but it appears to be mostly focused on Apple Intelligence.
I have also read that if you develop your prompt telling AI to only use certain resources, it is less likely to hallucinate. In short, there are ways to do better, but it takes actual work and practice and you still need to be careful and check, so unlikely to be used by anyone in this administration.
@Jc: Agreed.
A government report from our current government is lies, confabulations, with an occasional true statement mixed in to get you to swallow the bait. I don’t expect anything else, and you shouldn’t either.
RFK, Jr. is completely bought, signed in, and predictable. Remember when he ate that cheeseburger while on the airplane with Trump? That was the signal that he would swallow anything Trump does.
“This is the kind of thing that would lead an undergraduate to get a failing grade, and certainly this is not what one expects from the highest level of government. ”
Indeed. If this is all true we should all criticize it. But let’s not just wave away “whataboutism.” We had a vegetable as President. All the important people knew. It was a breach of trust larger than any in my lifetime.
All credibility is lost if criticism only goes in one, partisan, way.
The last big government report with blatant falsehoods resulted in a multi-year quagmire and hundreds of thousands of deaths in Iraq.
This can potentially be much worse.
@Connor:
The kind of lie that explains why Tapper’s ratings crashed. This is video of Biden in October 2024. Clearly not a "vegetable."
The issues surrounding Biden’s real physical aging are troubling enough, without the patriarchy’s hyperbole.
So why are you changing the subject instead of criticizing it?
I recall you defending DOGE cuts by claiming hard fiscal decisions were needed. Trump and Republicans are now pushing $4 trillion dollar deficits that dwarf any savings from those cuts. Where is your criticism of this fiscal insanity?
Nonexistent.
Where’s your criticism of Trump’s relationship with Epstein and his public sexualization of his daughter? Where’s your criticism of his call for Hitlerian generals? Where’s your criticism of his Jan 6 terror attack? Or his constant lying and his unhinged, unpresidential, serial killer style rambling on Truth Social? Of his tariff chaos, Islamic bribe planes, and crypto self-dealing?
Where’s your criticism of his unconstitutional attacks on 5th Amendment due process?
Nowhere.
Lecturing yourself? The hosts and commentariat here clash frequently over competing critiques of Biden, Harris, and Democrats/liberals.
MAGA 2.0 is a firehose of incompetence, corruption, and anti-conservative apostasy. The Trump troll response is invariably tortured excuses, half-baked apologetics, or — in particularly egregious cases — conspicuous silence.
Credibility is never had in the first place when y’all are blatantly hypocritical pot-stirring prevaricators not holding yourself to your own fake standards. You ain’t foolin nobody but yourself.
@steve: “Harvard prof didnt properly cite some stuff and used some stuff out of Wikipedia, the same as Altman’s wife, who is still a professor.”
Taking nothing away from your comment, I believe you meant Ackman’s wife. Altman is the AI guy, so he has a place in this conversation, but Ackman is the billionaire hedge fund manager and Harvard alum with the professor wife who ended up apologizing for some of the problems with her dissertation.
Ackman seems to harbor a certain kind of resentment toward Harvard. Not the resentment of students, and their families, who have little to no chance of being admitted to an elite school. His resentment is that of a benefactor whose donated millions do not buy the influence he’d like to have over the school. Perhaps it’s philosophical differences, as he’s indicated, or maybe he’s found that he can’t sufficiently influence the admission of friends and family members, or maybe he just takes issue with the 21st century policies on need-blind admissions and financial aid to students with demonstrated need, and thinks they should go back to the policies of his era.
@Connor:
It is exactly whataboutism because you are avoiding talking about what is being discussed.
You want to talk about other things, there is a daily open forum wherein you can raise it daily, if you so desire. Further, there have been posts on the topic.
If you can’t manage to comment on the topic at hand, please don’t comment.
You are also telling on yourself if you can’t see the problem with what the OP underscores.
You continually show yourself to be unserious and that you appear to prefer being an irritant to being a participant.