The media over the last couple of days has been pointing to a revised paper by Abhay Aneja, John J. Donohue III, and Alexandria Zhang that has been floating around in various forms since at least 2009. Their earlier paper claimed that right-to-carry laws and no impact on murder, rape and robber. While much the earlier versions of the paper remained unchanged, the authors add a few more years of data for some of their estimates. Some of the changed estimates are done only using later years and not the whole sample. On Friday, Christopher Ingraham in the Washington Post felt this unpublished “new research debunks a central thesis of the gun rights movement” of “More Guns, Less Crime” (see also discussions in the Huffington Post, the Raw Story, and other newspapers around the country that picked up the claim).
There are many errors in Ingraham’s article. For example, “Stanford law professor John Donohue and his colleagues have added another full decade to the analysis.” Yet, the third edition of “More Guns, Less Crime” has data from 1977 to 2005. Moody, Marvell, Zimmerman, and Alemante have a new paper earlier this year that looked at data from 1977 to 2006. Gius (2014) looked at data up through 2009. Zimmerman (2014) looks at crime data up through 2010. And note that none of those papers agree with Aneja, Donohue, and Zhang’s conclusion. Previously even in the Washington Post, Emily Badger’s misleading column also discussed an earlier version of Donohue’s paper with data through 2006 (7/29).
1) The problem with using the 1999 to 2010 period of time is discussed extensively in this short paper available here (download recommended). The abstract reads as follows: “Unfortunately, many who have examined the impact of so-called “shall-issue” or “right-to-carry” laws assume that the adoption of such laws causes a large, immediate increase in the number of permits. But that is often not the case, for states differ widely as to how easily permits can be obtained. This problem is particularly problematic for studies that have looked at the period after 2000. In fact, the share of the adult population with permits increased less during the 1999-2010 period in the states that adopted right-to-carry laws than the states that they are being compared against.”
2) Take a big claim in the Aneja, Donohue, and Zhang paper:
If we estimate both the dummy and spline models using our preferred specification without state trends for each of these two time periods (overall or after 1999), then we have 4 estimates of the impact of RTC laws for each of seven crime categories (Tables 8a and 11a). In each of the seven crime categories, at least one of these four estimates suggests that RTC laws increase crime at the .10 level of significance, with murder, rape, and larceny estimates reaching significance at the .05 level. These crime increases are substantial, with the dummy variable model for the complete period (Table 8a) suggesting that RTC laws increased every crime category by at least 8 percent, except murder (in that model, murder rose 3 percent but it is not statistically significant). For the post- 1999 regressions, spline estimate (Table 11a) suggests that RTC laws increased the rate of murder by 1.5 percentage points each year (significant at the .05 level). In none of those 28 regressions was there any statistically significant estimate suggesting that RTC laws decreased crime.
Thus, the evidence that RTC laws increase crime is strongest if one accepts the dummy variable model with our preferred specification on state data (the Table 8a and 11a results) and accepts the Wolfers (2006) critique that one should avoid controlling for state trends.
The two sets of estimates can be seen here.
In Table 8a, the Aneja, Donohue, and Zhang now argue that a simply “dummy” variable that measures the before-and-after crime rate is their ideal specification. In Table 11a, they point to a comparison of before-and-after trends.
1) Ironically, Donohue has had a number of papers arguing that the simple “dummy” variable method is misleading. For example, see the discussion in Ayres and Donohue, Stanford Law Review, 2003.
Here is a discussion of the general problem of relying on just before-and-after averages as measured by a simple “dummy” variable (second edition of More Guns, Less Crime in 2000 (see also the same point on pp. 1219-1220 in Ayres and Donohue (2003))).
3) Despite these flawed estimates that are biased towards finding the results that Aneja, Donohue, and Zhang want, the large majority of their own results don’t show any harm from concealed carry laws. ADZ claim aggravated assaults provided by far the strongest and consistent support for their claim, but then note even then just 11 of their 28 estimates show an increase in crime and these are the primarily the results just discussed in points (1) and (2). From ADZ’s abstract:
4) Aneja, Donohue, and Zhang falsely claim over and over again that previous research failed to account for crack cocaine. Hypocritically, they then fail to actually use this data and don’t even note that Lott used the very data they recommend for all the years that it is available in the third edition of “More Guns, Less Crime.” ADZ argue that estimates that don’t account for cocaine “are marred by omitted variable bias,” but then, unlike John Lott’s work, none of ADZ’s regression estimates actually account for cocaine.
Now here are the results graphically from “More Guns, Less Crime” (third edition, 2010) that used the data that ADZ say should be used for the years that they say it should be used.
Regarding committed variable bias, it is also interesting to note, that Donohue in all of his papers, including ADZ, refuses to account for any other types of gun control laws. The reason is pretty obvious as he realizes that accounting for any other gun control law in the types of estimates that he runs would not provide support for them. By contrast, Lott’s “More Guns, Less Crime” accounts for 13 different types of gun control laws to test the sensitivity of the right-to-carry laws. At least implicitly, these authors are arguing that none of these other gun control variables are important in explaining crime rates. But Donohue has strongly come out publicly in favor of other gun control laws. There is no problem with him supporting other gun control laws, but if he really believes that they are useful, not including them implies that his estimates here are misspecified.
5) These media also don’t put this discussion in any type of context. Not even including refereed research by Moody, Marvell, Zimmerman, and Alemante (2014), Moody, Lott, and Marvell (2013), and Gius (2014) who claim that right-to-carry laws reduce violent crime and Zimmerman (2014) who claims no effect but gets the interpretation of his results backwards (see here and it is my understanding that Zimmerman agrees with this), the vast majority of peer-reviewed research finds that right-to-carry laws reduce violent crime.
6) The earlier versions of this paper had serious data and specification errors in them. A 2012 paper by Carlisle E. Moody, John R. Lott, Jr., Thomas B. Marvell, and Paul R. Zimmerman found the following errors:
• The first full year of the shall-issue law for Kansas is coded as 1996 when in fact the law was not passed until 2006.• The first full year of the shall-issue law for Florida is coded as 1989 when in fact the law was passed in 1987.• The first full year of the shall-issue law for South Dakota is coded as 1987, however the law was passed in 1985.
• includes 100-140 more counties each year that should have been removed from the set of 3,140 counties because of errors in their reporting.
. . . the errors in ADZ’s data set biased the results towards what they wanted to show. . . .