Earlier today, James wrote a post exploring the reporting (or claimed lack-there-of) of the latest quarterly Uniform Crime Report (UCR) data issued by the FBI. As many criminal legal system* experts predicted, overall violent crime (and, in particular, the murder rate) is continuing to trend down towards their pre-pandemic levels. In response to this, one of our regular contrarian commenters posted the following:
Of course, that is headline crime and only what the cities and state report. But in many urban areas, police don’t respond to crimes. I was reading an article about the dead zone downtown Minneapolis was now. The author of a quoted article reported that police didn’t respond when his daughter was mugged as no one was injured. That crime isn’t going to be in the statistics, but it will be in the local gossip.
Report the crime stats are down and get the comments filled with anecdotes. Accurate or not, the comments will reflect the opinion of the citizens.
This concern about under-reporting is very familiar to those of us with experience in the criminal legal system data space. It’s especially common to see this argument deployed anytime URC data shows the crime rate going down–typically for law-and-order political reasons. With the help of an expert on the topic, I want to take a moment and unpack why this line of thinking doesn’t hold much weight.
To unpack this topic, we first need a little grounding:
Criminal legal system data in the US sucks To use a technical term, US criminal legal system data (especially when compared to other developed nations) is “the liquid shits worse.” If you think medical data and education data are bad, they are lightyears ahead of criminal legal system data. There are many reasons for this including the fragmented and opaque nature of our criminal legal system (I’m going to use CLS), the lack of unified recording systems, the reluctance of people to give or record accurate data, and the fact that different case management systems ask for different data. It’s so bad that there are a number of nonprofits, including my former employer Measures for Justice, dedicated to trying to collect and format data into a comparable form.
UCR data has always been incomplete Participation in the UCR is voluntary. Each arresting agency in the United States, and are approximately 18,000+ of them, must choose to opt into those services. While there are incentives to participate (e.g., access to federal programs), not all agencies choose to. That has been the case since the beginning of the program in 1929.
Under-reporting is real and happening today We know that crime is, and more importantly, has always been, underreported. This topic has been explored in the CLS and Economics fields** for decades. For example, see the seminal paper “Why are crimes underreported? What is the crime rate? Does it really matter?” by S L Myers, published forty years ago in Social Sciences Quarterly. And for as long as this question has existed, there have been many attempts (using different methodologies) to quantify this level of under-reporting.
Part of the challenge is there can be many reasons for underreporting. We know that historically sexual assault has been underreporting for a wide number of reasons. Domestic abuse is often underreported out of a desire to protect the offending partner. Likewise, various violent and non-violent crimes go unreported to protect friends and family members. Finally, a not insignificant amount of crimes are tied to other crimes. For example, if you are engaged in dealing drugs and someone steals your supply or your income from drug sales, chances are (if you are smart) you are not going to turn to the police for help.
Given all that, why can we ever trust UCR data? The quick answer is you can’t–at least as a snapshot. However, no data are perfect, and for a number of reasons, when we look at the data set in aggregate, it becomes very reliable for understanding things like trends. Thankfully, you don’t have to take my word for it. As I was prepping to write this, I discovered an excellent Substack post on the topic by Jeff Asher. I got to meet Jeff during my time at Measure for Justice. He’s a very careful data analyst with a background in CLS data. Jeff also has a knack for explaining things. If you have the time I highly recommend reading the article, Is Crime Underreporting Getting Worse?, in full. Unsurprisingly, he was responding to exactly the same sort of question that triggered this post.
Jeff, smartly, sees three key questions at the heart of this topic:
The question above isn’t one question but three different questions that are worth addressing. The three questions are:
Is crime underreporting getting worse?
If underreporting is getting worse, should that make us question the overall national trend?
Are prosecutors to blame? [note this isn’t as useful for this discussion-mb]
And my answers are “maybe, maybe not, we don’t know”, “probably not”, and “I’m pretty skeptical.”
Q1. Is crime underreporting getting worse? As you can suspect from his answer of “maybe, maybe not, we don’t know” the ultimate honest answer is we cannot know for sure.
Drawing on his experience working with police departments on their data, Jeff takes us through an important thought exercise. First, he acknowledges that we have data that show the size of certain metropolitan police forces has decreased. Those decreases have been correlated with longer response times. However, Jeff points out, looking at specific cities doesn’t tell the whole story:
Seattle is not alone in losing officers as most cities above 100,000 people lost officers between 2019 and 2022. Conversely, most small cities and counties gained officers between 2019 and 2022. […]
So while worsening underreporting in 2023 in some agencies may be balanced out by other agencies seeing less underreporting this year (along with many agencies where it wasn’t a problem to begin with). Unfortunately, high quality publicly available Calls for Service data is hard to come by which severely limits our ability to understand whether underreporting got worse in large chunks of the United States or if it was contained in a few extreme outlier examples.
Of course response times aren’t the only reason property crimes don’t get reported (to say nothing of the complex reasons for systemic underreporting of certain types of violent crimes like sex offenses). Many stores don’t report small-scale shoplifting offenses by matter of policy. One would think that, given the increased attention to the issue of “organized retail crime”, that the share of such offenses that are reported by stores would increase rather than decrease.
I am sure the contrarians reading this will think, “Ah ha! You can’t prove underreporting *isn’t* getting worse!” Jeff and I would agree and also remind them, “True, but you cannot prove it’s getting better.” Further, I would say (as I cannot speak for Jeff) that underreporting, being a long-standing issue, is already baked into the data and any careful analysis of it.
Jeff takes a different approach in his analysis:
Q2: If underreporting is getting worse, should that make us question the overall trend? He chooses, as part of his analysis, to temporarily accept the premise that underreporting IS getting worse. Jeff then goes on to offer three reasons why the declines in the data are still valid:
First, murder is down huge and we know that murder is not underreported. The crime we know to be the most accurately reported is plunging which means other types of crime falling is at least plausible. Reported violent crime trends are pretty strongly correlated with murder trends nationally from year to year, so the fact that murder is down quite large reinforces the likelihood that overall violent crime would follow a similar trend. Murder and property crime are less strongly correlated but they have tended to move in the same direction — more or less — since 1960. […]
Second, the declines shown in the preliminary FBI quarterly data are quite large outside of auto theft and the impact of underreporting is likely to be reasonably small in the grand scheme of things when calculating the year-on-year change for 2022 vs 2023. […]
Third and finally, remember that Seattle is most likely an outlier here and most underreporting is going to be occurring in larger cities of 100,000 people or more that have lost officers. The change in crime as suggested by the preliminary quarterly data, however, is very large pretty much everywhere you look. Indeed, the decline in property crime nationwide would be improved if you removed bigger cities that are more likely to have larger surges in auto thefts this year.
Then, returning to the broader question of whether underreporting is happening or not, Jeff gives one of the best answers I have seen on the topic: “We can’t know for certain whether underreporting is actually worsening. The available evidence points to possible problems in some places with possible improvements elsewhere, which implies any impact on national trends is likely minor.”
Q3: Are prosecutors to blame? For those involved in the CLS reform space, these attacks on URC data are most normally trotted out to attack reforms (in particular) “progressive prosecution” strategies. To avoid making a long post longer, I recommend you read that section. Regardless of the local prosecution approach, the fact that we are seeing an across-the-board decline is important when judging policy impacts. Rates are falling at similar rates in places with “tough-on-crime” prosecutors and “progressive” prosecutors. Also, let’s not forget, as Radley Balko likes to remind us, that the “high crime rates” also grew in 2020-2 under those “tough-on-crime” prosecutors as well.
Jeff’s conclusion (and mine too for part 1) Jeff gets to the heart of the issue in his excellent summation:
To finally answer the initial question, I would say that the changing crime trends shown in the FBI preliminary data are preliminary and undoubtedly imprecise but likely reasonably accurate. They are imprecise because the year is not complete, the reporting is not finalized, and only around 80 percent of the US population was represented in their data. The data we get 10 months from now may show smaller decreases or even small increases and we shouldn’t be shocked to see that because of the preliminary nature of the data.
The problem with automatically claiming the national crime trend is getting worse solely because underreporting is worsening is that it creates a situation where the trend can never go down.
It’s already problematic to base our understanding of crime trends on anecdotal readings of reported incidents (I think crime is up because I saw a bunch of robberies reported on the news). It’s much worse — in my opinion — to base one’s impression of the trend on anectdotal readings of unreported incidents (I heard about some incidents that didn’t get covered so it can’t possibly be getting better).
I can’t personally have said it better–especially that last paragraph that speaks to this portion of the contrarian viewpoint:
Report the crime stats are down and get the comments filled with anecdotes. Accurate or not, the comments will reflect the opinion of the citizens.
I’ll deal with this portion in a post that I expect to get out tomorrow. For the moment, I want to point out an ironic point about the use and dismissal of UCR statistics, in particular by people who tend to be on the political right:
It’s a truism that for many, UCR trends are always trustworthy when they show crime going up and always untrustworthy when they show it going down.
I hope that intellectual curiosity will lead people to notice that pattern (either through personal introspection or when it’s pointed out to them) and get interested in exploring why they think that way.
Tomorrow, I’ll dig into some of the cognitive biases that prevent that from happening.
*—I use “criminal legal system” rather than “criminal justice system” because our system of laws is optimized to keep itself working rather than actually delivering any abstract philosophical concept of justice.
** – I realize that some contrarians, despite often citing the writings of academics they agree with, typically dismiss anyone who has degraded themselves to the level of teaching in a higher education setting to be completely committed to ending Western Civilization through radical leftist indoctrination. I just wanted to point out that Criminology and Economics, two fields that often go together, have some of the highest rates of self-identified Republican and Conservative scholars in the Social Sciences. In fact, many Criminologists were law enforcement officers*** who went into academics.
*** – I realize that being a former law enforcement officer is immediately disqualifying to other contrarian commenters. Likewise suggesting that any prosecutor could be “progressive.” What can I say, I can’t please everyone.
Matt Bernius is a design researcher working to create more equitable government systems and experiences. He's currently a Principal User Researcher on Code for America's "GetCalFresh" program, helping people apply for SNAP food benefits in California. Prior to joining CfA, he worked at Measures for Justice and at Effective, a UX agency. Matt has an MA from the University of Chicago.
Nice. This is how you should think about this kind of data. The absolute number may be wrong but the trend still gives you important information. The downside to this is that folks on the right will look at this accurate nuanced statement and say “See, they dont know and dont want to admit crime is increasing”.
I mentioned before how much of this is driven by motivated reasoning.
“The world is changing in ways I don’t like, so therefore crime/ the economy/etc. must be getting worse.”
This isn’t the only cause- the standard tendency to hear only about planes that crash is potent as well. But it is remarkable how committed conservatives are to the Decline narrative.
“It is a truism that…”
Exactly!Thanks for the lucid discussion.
BTW, if you are interested in Jeff Asher’s grounded thoughts on the latest data release, there’s a point for that. Its good and as always he shows his work:
https://jasher.substack.com/p/fbi-quarterly-data-points-to-declining
Great article, Matt. Love your content and your takes.
(But I still love you too, James and Steven…)
Kudos on the article.
When I’m looking to move somewhere the only crime stat I trust is the murder number. It’s hard to rig murder numbers, except on The Wire.
Nice. This is how you should think about this kind of data. The absolute number may be wrong but the trend still gives you important information. The downside to this is that folks on the right will look at this accurate nuanced statement and say “See, they dont know and dont want to admit crime is increasing”.
Steve
I mentioned before how much of this is driven by motivated reasoning.
“The world is changing in ways I don’t like, so therefore crime/ the economy/etc. must be getting worse.”
This isn’t the only cause- the standard tendency to hear only about planes that crash is potent as well. But it is remarkable how committed conservatives are to the Decline narrative.