Live Science describes a new study in the American Journal of Preventive Medicine this way:
Monuteaux and his colleagues wanted to test whether increased gun ownership had any effect on gun homicides, overall homicides and violent gun crimes. They chose firearm robbery and assault, because those crimes are likely to be reported and recorded in the Federal Bureau of Investigation (FBI) Uniform Crime Report.
Along with that FBI data, the researchers gathered gun ownership rates from surveys in the CDC’s Behavioral Risk Factor Surveillance System, an ongoing, nationally representative survey in which participants answered questions about gun ownership in 2001, 2002 and 2004. Using those years and controlling for a slate of demographic factors, from median household income, population density, to age, race and more, the researchers compared crime rates and gun ownership levels state by state.
They found no evidence that states with more households with guns led to timid criminals. In fact, firearm assaults were 6.8 times more common in states with the most guns versus states with the least. Firearm robbery increased with every increase in gun ownership except in the very highest quintile of gun-owning states (the difference in that cluster was not statistically significant). Firearm homicide was 2.8 times more common in states with the most guns versus states with the least. . . .
But their paper (available here) isn’t testing what they claim. It isn’t testing whether increased gun ownership causes crime rates to increase. The study is far too simplistic and doesn’t include even the basic control variables that are typically included in other crime studies.
Here is a simple example. Many people point to the fact that the UK has both a lower homicide and gun ownership rate than the United States. The claim is often made then that the reason that they have a lower homicide rate is because they have fewer guns. However, this ignores the fact that the UK homicide rate actually went up after their 1997 handgun ban or after their other very strict earlier gun control regulations. The UK homicide rate still remained low relative to the US, but it was higher than it otherwise would have been.
The point here is simple: there are lots of reasons why the UK homicide rate was lower than that in the US before they even had gun control. The question that needs to be asked is how the UK homicide rate changed relative to that in the US after its gun control regulations went into effect. To do this, you have to control for the fact that the UK had a much lower homicide rate to begin with. Statistically you do that by having what are called geographic “fixed effects” (dummy variables that pick up the average difference in each jurisdiction that you are examining). Any test would also do the same thing by year so as to account for any national trends in crime rates. So, for example, crime might have been falling nationally, but was it falling relatively more in those states that were getting more gun ownership.
The controls that are being used in this paper can’t begin to account for the differences in crime rates. The regression estimates reported in Table 2 don’t tell what percent of the variation in crime rates are being explained by the variables used in these regressions, but I am willing to bet that it is less than 10 percent.
Yet, this paper in the American Journal of Preventive Medicine doesn’t account for either of these factors. It is essentially making a purely cross-sectional comparison across states. On account of that, if they had included Washington, DC in their estimates (with its high crime rates and low gun ownership), it would have dramatically altered their results.
It is easy to see how the results are reversed by just including these state fixed effects. The first estimate below corresponds to the first estimate reported in Monuteaux, Lee, Hemenway, Mannix, Fleegler’s paper. The first estimate uses regional fixed effects. The second includes state fixed effects. The gunBRFSS variable is their survey measure of gun ownership by state. In order to get at the nonlinear concern that they raise, I do want academics normally do and have both a linear and a squared version of that variable. You can see that including the state fixed effects causes their result to go from positive and insignificant to negative and significant. It seems clear that they broke the survey measurement into arbitrary fifths to help get the result that they wanted.
We used the negative binomial approach used by these authors in these estimates, but there is no truncation issues here and the data fits a weighted least squares estimate. However, just for the sake of argument we will use the approach that they want used. (Click on results below to enlarge them.)
While these “fixed effects” will pick up the average differences across places, there are other differences that won’t be accounted for. One example is they don’t account for differences in any type of law enforcement (e.g., arrest or conviction rates, death penalty, per capita number of police, percent of the population in prison). The above results include the arrest rates for aggravated assaults, but removing the arrest reduces the statistical significance for both estimates.
The authors ignore previous refereed published research on this in More Guns, Less Crime (University of Chicago Press, 2010) (see discussion here). In those estimates state level fixed effects are used to pick up the average difference in crime rates. Possibly citing this research would have forced the authors to explain why they got such difference results.
Finally, there is also the issue of whether people in high crime areas are more likely to get guns for protection. The question is whether increased gun ownership is caused by higher crime rates or the reverse.
Here are the results with weighted least squares.
Here is what happens if you use weighted least squares and a linear version of the percent of the population with guns and no state fixed effects.