The end of last week a study in the American Journal of Public Health claimed that there were more police feloniously killed in states that had more guns. The study got extensive news coverage on the TV networks such as NBC News, dozens of newspapers such as the Chicago Sun-Times, and international coverage such the UK Guardian. Yet, it took just a couple of minutes to read the paper and realize that the empirical work was done in a very non-standard way. There is a big benefit to using so-called panel data, where you can more accurately account for differences in crime rates across states or over time. This method is called “fixed effects.” Ask any academic who deals with this type of data, and they will tell you that these are basic controls that all papers in this area account for. Strangely the authors, David Swedler, Molly M. Simmons, Francesca Dominici, and David Hemenway, only control for the differences across states and not over time.
A couple of simple examples show why other studies on crime take into account these factors
Take the differences across places. Many people point out that the UK has both a lower gun ownership rate and a lower homicide rate than the US. Some use this to claim that gun control causes crime rates to fall. But the homicide rate actually went up by 50 percent in the eight years after the 1997 handgun ban went into effect. The homicide rate was still lower than that in the US, but there were lots of reasons it was lower to begin with, not the handgun ban.
The same point applies over time. Suppose a state passes a gun control law at the same time that crime rates are rising nationally. It would be a mistake to attribute the overall increase in national crime rates to the law that got passed. To account for that concern, researchers normally see whether the increase in crime rate for the state that had the change is greater or less than the overall national change.
Unfortunately, the American Journal of Public Health study doesn’t account for this last concern, and their results are extremely sensitive to this error. Here is a simple version of their regression explaining the number of police officers feloniously killed over the years from 1996 to 2013, but without accounting for the time effects that we just discussed. Before going through the results, the media coverage of this study is incorrect in claiming that more guns are associated with more police killings. What the journal article actually measures is not the gun ownership, but the percentage of suicides committed with guns (fsdsuicides). We have expressed our concerns about this measure many times, but since these authors and many other academic authors use this measure and the media is reporting this as gun ownership, we will use it here primarily for comparison purposes. This regression looks at the total number of police feloniously murdered with their measure of “gun ownership” and the number of police officers. In this regression, it appears that a one percentage point increase in the percent of suicides committed with guns increase, there is a significant 1.8 percent increase in the total number of police killed.
It seems hard to believe that Swedler, Simmons, Dominici, and Hemenway wouldn’t have tried accounting for these differences as their first estimates on explaining police deaths. The fact that they never mention how this alters their results is troubling. As usual, the AJPH was not interested in publishing a comment pointing out this sensitivity. For any media that might be reading this discussion, let us suggest that you contact an empirical economist or criminologist who deals with panel data of your choosing and ask them these simple questions. If you are dealing with panel data, how unusual is it for the authors of a peer-reviewed paper not to ever account for these changing differences over time? If they didn’t somehow account for this factor any place in their paper but doing so caused you to get the opposite results from what they claim, what would that do to your evaluation of their conclusions?
Just accounting for the changes in crime rates over time, completely reverses the claims made in the American Journal of Public Health. It now appears that a one percentage point increase in the percent of suicides committed with guns increase, there is a significant 2.25 percent decrease in the total number of police killed.
Including the other variables that they have used in their estimates along with the year fixed effects produce an even more statistically significant drop in police felonious killings. In that case, a one percentage point increase in the percent of suicides committed with guns increase, there is a significant 3.62 percent decrease in the total number of police killed.