Outside the Beltway

Why We Can Say Crime Is Down Despite The FBI Upward Revision 2021’s Crime Estimates

Ignorance and motivated reasoning are never a good combination

Matt Bernius · Tuesday, October 22, 2024 · 19 comments

Over the weekend, one of our posters tipped me off to one of the latest conservative media complex tempests in a teapot: the FBI revising its 2021 estimate of crime significantly upwards. This story, broken by the conservative news site Real Clear Investigations (part of the Real Clear Politics family), was quickly picked up by Newsmax and other right-wing websites as used as “proof” that the FBI is cooking the books to hurt Donald Trump’s reelection chances. We saw slightly different versions of this claim in comments yesterday and this morning as well. The question we should be asking is: is it accurate?

Since criminal legal system data and statistics is one of my areas of interest, I researched the topic and the results of my investigation (will not) shock you! Here’s the tl;dr version in bullet form:

• Yes, the FBI revised the 2021 numbers significantly upwards.
• Contrary to what an “expert” says, data revisions like this are actually routine.
• This wasn’t a surprise for anyone paying attention to these data as the 2021 numbers were known to be off from the start.
• Trends are determined by looking across years and data sets, and all the evidence still shows that overall crime is down from its peak and continuing to fall.

The rest of this post will explain how Real Clear Investigations spin is a mix of falsehoods, facts without context, and the old wine of “you can only trust the data when crime goes up, but not when it goes down… especially if you don’t like the party in power.”

Along the way, I’ll touch on how (willful) ignorance, the Dunning Krueger Effect, and motivated reasoning effect the way folks think about this type of data. And, as a gift to the “you can never trust the FBI” folks, I will also critique the agency and the Bureau of Justice Statistics for a lack of transparency that doesn’t help the situation.

The claim

The Real Clear Politics Investigation states:

When the FBI originally released the “final” crime data for 2022 in September 2023, it reported that the nation’s violent crime rate fell by 2.1%. This quickly became, and remains, a Democratic Party talking point to counter Donald Trump’s claims of soaring crime.

But the FBI has quietly revised those numbers, releasing new data that shows violent crime increased in 2022 by 4.5%. The new data includes thousands more murders, rapes, robberies, and aggravated assaults.

All of that is correct. However, as we will see, it leaves out the important context: the issue isn’t so much with the 2022 or 2023 data but the 2021 data, more on that in a bit. However, one thing to note is the scare quoting of “final” in that first sentence. Looking at the 2023 press release associated with that release, there is no indication that the data set being released was the final version. Anyone who works with those data sets knows that, as with most government data sets, they are often updated a few times after the initial release.

In fact, the criminologist that Real Clear Investigations cites points out as much (though in a way that is phrased to make it appear that this practice isn’t routine and also gets the facts wrong):

“I have checked the data on total violent crime from 2004 to 2022,” Carl Moody, a professor at the College of William & Mary who specializes in studying crime, told RealClearInvestigations. “There were no revisions from 2004 to 2015, and from 2016 to 2020, there were small changes of less than one percentage point. The huge changes in 2021 and 2022, especially without an explanation, make it difficult to trust the FBI data.”

Here’s the first issue with the RCI article: Moody is flat wrong that the UCR wasn’t updated before 2016. As Jeff Ascher points out in one of his essays on this topic:

Here’s the 2006 report — see footnote 3 — noting that the 2005 figures were revised. And here’s 1995’s report (see footnote 4 on page 58).

So if Moody isn’t familiar enough with this topic to know this, perhaps he’s not the best person to comment on the UCR data sets. And frankly, if Real Clear Investigations was interested in representing facts, they should have fact-checked that claim.

For the sake of argument, let’s say he was right and the practice only began in 2016 (again this is NOT the case). That would still mean that the practice of issuing revisions has been with us for almost a decade, since the start of the Trump administration in fact. Also note that the pattern changes in 2021… it’s almost as if something systemic might have happened in 2021. We’ll get back to that. We’ll also address the idea that this happened “without explanation” in the next portion of this post.

This get’s us to the second issue with the article, it’s framing that something nefarious is going on. Again, from Jeff Asher:

Calling these changes “stealth” simply displays one’s lack of awareness that revisions are frequently made to historical FBI estimates. The FBI doesn’t do press releases about the UCR’s methodology, it’s frustrating to those who deal regularly in crime data (at least it is to me!) but it doesn’t imply some malice or effort to hide the revisions.

In other words, the FBI has been following the same release procedures it’s been using since at least the Trump administration (if not long before). I think there is definitely a discussion to have about their level of transparency, but don’t pretend they are actively hiding something (which is a great example of motivated reasoning getting in the way of the facts).

From there, the RCI article then moves on to detailing the changes within the data set. Since I’m not contesting those changes, I will move over that part of the article.

Then we hit the third major issue from the article: the majority of it is a rehash of a key Trumpian argument that has nothing to do with the FBI’s data set, that crime is really up because crimes are not being reported. For reasons I’ve already covered, this argument doesn’t stand up to scrutiny and when you look holistically at all crime-related datasets (not just the FBI’s Uniform Crime Reporting database), crime continues to be trending down.

So, what happened with the FBI data sets?

While I am sure that Dr Moody is an expert in his area of criminology (which seems to be primarily focused on the relationship between gun ownership and crime rates), his statement suggests that he either hasn’t been following discussions about changes to the FBI’s UCR data closely or he’s allowing his conservative leanings (and having reviewed his papers, those leanings are pretty apparent) lead him to feign ignorance about some well-known facts around changes to the dataset.

Once again, I turn to policing and crime data expert Jeff Asher for an evenhanded explanation (which includes some fair critique of the FBI). Jeff’s first article on this specific topic begins with this important grounding:

The FBI publishes [crime] estimates each year. They’re estimates because not every agency reports data every year. Anybody suggesting the FBI’s figures are perfect is either naive about the way the data is collected and reported or doesn’t read my newsletter (or probably both). But the FBI’s figures, much like the BJS figures, paint a comprehensive picture of American crime trends over time even if the exact numbers can never be 100 percent correct.

The FBI’s 2021 crime estimates are the exception. The 2021 figures were not reliable when they were first published in 2022 and they are not reliable now.

Around 65 percent of the country was covered by a NIBRS agency when the switch was made in 2021. It’s why the FBI’s 2021 crime report came with very large error bars that were much maligned.

Participation for 2021 rose to around 73 percent when the 2022 estimates were published as a handful of non-NIBRS places like New Orleans added their 2021 totals when they reported 2022, and the 2021 participation rose again to 74.1 percent when 2023’s figures were reported. Some agencies have reported 2021 data to the FBI in later years, but NYPD and LAPD have not while Chicago has reported only a few months of 2021 data. Also not reporting in 2021: Phoenix, Miami-Dade, San Jose, San Francisco, Raleigh, Omaha, Oakland, Pittsburgh, and I could go on.

So, what is NIBRS? From its FBI page:

[The National Incident-Based Reporting System (NIBRS) was] implemented to improve the overall quality of crime data collected by law enforcement, NIBRS captures details on each single crime incident—as well as on separate offenses within the same incident—including information on victims, known offenders, relationships between victims and offenders, arrestees, and property involved in crimes.

Unlike data reported through the UCR Program’s traditional Summary Reporting System (SRS)—an aggregate monthly tally of crimes—NIBRS goes much deeper because of its ability to provide circumstances and context for crimes like location, time of day, and whether the incident was cleared.

As recommended by professional law enforcement organizations, the FBI has made nationwide implementation of NIBRS a top priority because NIBRS can provide more useful statistics to promote constructive discussion, measured planning, and informed policing. To increase participation, the UCR Program is partnering with the Bureau of Justice Statistics on the National Crime Statistics Exchange, working with advocacy groups to emphasize the importance of NIBRS data, and transitioned the UCR Program to a NIBRS-only data collection, as of Jan. 1, 2021.

For the record, it’s possible (though highly unlikely) that Dr Moody was unaware of this transition and its ramifications. Or he might have included that in his interview, but the RCI journalists chose not to include it. Either way, it’s a major missing piece of context, which along with his false statement about revisions leads me to question his authority to speak on the UCR. Especially as this is a topic that has been long discussed in the criminal legal system data space:

So the FBI’s estimates for 2021 still are not good which is why I wrote last month:

“Also note that the 2021 estimates were based on substantially lower participation due to the NIBRS switch, so I’m not comparing it to other years before or after in this analysis. The 2021 figures are included in graphs for the sake of continuity though they can be largely ignored.”

And in July 2023 I wrote:

“The national crime estimates for 2021 will likely always require an asterisk, but one skewed year of crime statistics is much easier to accept than three or four skewed years.”

Our assessment of whether crime is rising or falling really shouldn’t include the FBI’s 2021 estimates.

That last sentence is particularly critical, because that 2021 data set is the one that RCI is using to make it’s analysis that crime went up in 2022. This gets to a core issue that I, Asher, and many other people who are familiar with UCR data have: Conservatives find the data reliable when it confirms their priors and, simultaneously, unreliable when it contradicts them. And in the case of comparing 2022 to 2021, they are doing both at the same time:

People who — wrongly — argued for months about the FBI’s lower participation in 2022 and 2023 really shouldn’t be suddenly relying on 2021’s estimates as an accurate portrayal of crime nationally that year.

For the record, that argument (and contraction) constitutes the entire back half of the RCI article.

Why the revisions?

This is where Asher and I both think the FBI should be more transparent. We know that some of the reasons in recent years can include more police departments getting back data onto the NIBRS and those new data affecting the estimates. However, that isn’t the full explanation:

The revisions to 2023’s figures were large by historical standards though this isn’t the first time we’ve seen somewhat sizable revisions to previous years. There were 825 murders added to the FBI’s originally published tallies for 2006 through 2009 by 2010, for example.

Why were the FBI’s revisions in 2022 and 2023 so large? I don’t really know which is frustrating. I asked about it last year when the 2022 revisions were published but didn’t get any answers back. Large revisions without an explanation isn’t a great practice, but it also wasn’t overly concerning for me personally because it only serves to highlight how these crime estimates should always be taken with a grain of salt.

This also gets to a second critique, this one directed at the press. As with all government data reporting, it should be emphasized that what’s being reported are estimates based on data, not immutable facts. They are a way to, over time, identify movement and trends. However, doing simple year-to-year analysis often leads to misleading perceptions (something that cuts both ways in the political space when talking about crime data).

Does this mean crime isn’t going down?

I’ll let Asher do the talking:

Do the 2022 and 2023 revisions majorly alter our understanding of national crime trends? In my opinion, not at all. …

If our assessment of crime trends rightfully ignores 2021 then we can say that violent crime rose slightly in 2020 while murder rose at the fastest rate ever recorded that year. That was true in the 2020 release, it was true despite revisions in 2022, and it was true in 2023. Again, ignoring 2021, then we can say that the FBI’s 2023 estimates show a continued small decline in violent crime with a historically large decline in murder.

The number of violent crimes in 2022 rose from 1,232,428 when initially published to 1,256,721 after revision, an increase of around 24,000. If you assume a similarly historically large revision will be applied to the 2023 data in 2024 (which I doubt will occur, but certainly can’t rule out) then violent crime still had a small down in 2023 relative to 2022 and was largely in line with the historically low violent crime rates observed from 2013 through 2019.

Asher’s second article on this topic goes really deep into the data in a way that defies easy excerpting. I recommend it to any data geeks and wonks out there to understand the issue better. For the moment, I want to share part of that article’s conclusion:

It’s certainly valid to want to understand why this change occurred and to hope for better transparency to help communicate the issue. A discussion on methodological transparency to better establish if/why the standard methodology was used for 2021 would be great. It’s frustrating! Welcome to the world of analyzing crime data from the outside!

The exact 2021 national crime rates are simply not knowable with any confidence. But the issue only impacts 2021, we were all supposed to ignore 2021’s estimates anyway, it is now 2024, and we have a multitude of sources [MB: including the Real-Time Crime Index, NORC’s Live Crime Tracker, the CDC, and the Gun Violence Archive] helping to articulate last year and this year’s crime trends which are fortunately very clear.

To paraphrase Hanlon’s razor, never ascribe to malice that which can be adequately explained by poor crime data.

There’s no conspiracy, there’s no attempt to deceive, there are not unprecedented stealth changes being suddenly made, the FBI didn’t suddenly “find” a ton of crime. There’s simply a methodology that’s poorly suited to lots of uncertainty being unclearly applied to a flawed year of crime data producing flawed, frustrating, uncertain results for that year and that year alone.

BUT THE FBI ESTIMATES CANNOT BE TRUSTED… HOW CAN WE KNOW CRIME IS GOING DOWN?

As a final flog on this particular dead horse, we can tell because the UCR isn’t the only data set people use to analyze crime trends. As noted in my quote revision above, the Real-Time Crime Index, NORC’s Live Crime Tracker, the CDC, and the Gun Violence Archive all show similar trends. So this is either a conspiracy of epic proportions or crime is continuing to go down after a spike related to a once-in-a-generation series of forcing events.

Unfortunately, as with most grounded arguments, this explanation is unlikely to overcome the potent cocktail of (willful) ignorance, Dunning Krueger (see Dr. Moody as an example), and motivated reasoning that has animated the “tough on crime” crowd for the last few years. On this that’s for sure though, any argument that we cannot trust the data for trend analysis is fundamentally based on “feels” rather than “facts.”