An unpublished study by John Donohue, Abhay Aneja, and Kyle Weber has received a lot of attention for supposedly finding some evidence that right-to-carry laws increase overall violent crime rates. It has been covered in Newsweek, The Atlantic, Bloomberg, Vice, Snopes, and many newspapers such as Newsday and the Salt Lake City Tribune. As is typical of Donohue’s work, there is no attempt to mention or respond to prior criticisms, and he just repeats the same, seriously biased methods and errors.
Publications such as Time and Newsweek would always interview critics when they ran stories on Lott’s original research. But when studies have the right political biases, reporters no longer get both sides of the story. But especially when the media explicitly describes a study as “debunking” John Lott’s previous research, you might think a reporter would call up Lott and get his take on it. It has been two weeks after the Donohue-led research started getting attention, and not a single reporter has contacted him.
Synthetic control tests were developed when panel data wasn’t available. Panel data allows you to account for lots of factors that are otherwise very difficult to account for. But the problem here is that this is panel data, so it makes no sense to use a second best approach which involves a lot of arbitrariness in the approach.
The bottom line is pretty clear: Since permit holders commit virtually no crimes, right-to-carry laws can’t increase violent crime rates. You can’t get the 1.5 to 20 percent increases in violent crime rates that a few of their estimates claim with only thousandths of one percent of permit holders committing violent crimes. To put it differently, states would have to be miss reporting 99%+ of crimes committed by permit holders for their results to be possible.
The synthetic control tests where they use anything from two to four states to predict the changes in another state’s violent crime rates are extremely arbitrary. For example, would you look almost exclusively to Hawaii to predict violent crime rate changes in Idaho, Minnesota, Mississippi, Nebraska, and Utah? Would you look almost exclusively at Illinois to predict changing violent crime rates in South Carolina? Remember that half of Illinois’ violent crime occurs in Chicago and an even larger majority of the changes in Illinois’ changing violent crime rate is due to Chicago. Would you look at California and New York to predict changing violent crime rates in Georgia?
There is a reason that the vast majority of published peer-reviewed studies that use US data as this new study does find that right-to-carry laws reduce violent crime rates. [UPDATE: There is a published paper by Carl Moody and Thomas Marvell that points out that even if you use the specifications Donohue, Aneja, and Weber prefer, their results are “fragile and most likely incorrect.”]
This new study looks at only murder and violent crime rates, and its most notable claim is that there is some evidence that violent crime rates rose after right-to-carry laws were adopted. But with over 60% of violent crimes involving aggravated assault, any changes in violent crime rates are be driven by changes in aggravated assaults. Other papers in this areas have looked at all the different types of violent crime (such as murder, rape, robbery and aggravated assaults), so it is quite unusual to only examine them as a group and it hides how other crime rates are changing. Donohue’s papers are the only ones to claim that there have been increases in aggravated assaults after right-to-carry laws were enacted. We will discuss the problems with this claim below. But more pertinently, permit holders commit virtually no aggravated assaults, especially not aggravated assaults with a weapon.
The authors’ purported increase in violent crime is driven by aggravated assaults. No explanation is offered for why having more permits would cause more assaults. No one would suggest that permit holders are more vulnerable to assault, so the claim would have to be that permit holders are doing the assaults themselves. But this is implausible, since permit holders are virtually never convicted of aggravated assaults, let alone aggravated assaults with firearms. Concealed handgun permit holders committing crime can simply not increase either aggravated assaults or violent crime.
Permit holders commit aggravated assaults and violent felonies at rates of thousandths of one percentage point, accounting for hundredths of a percent of the violent crimes or aggravated assaults committed in a state.
On page 45, the authors claim that “official withdrawals clearly underestimate criminality by permit holders,” but they offer no evidence for this claim. The provide one case in 2013 from the Huffington Post involving two permit holders who reportedly fatally shot each other. Another case from 2000 is provided, but that permit holder was prosecuted so it isn’t clear what this reference demonstrates. The point is clear: Even if convictions of permit holders are somehow being missed by reporting agencies, the error rate would have to be truly massive to explain Donohue, Aneja, and Weber’s results.
For example, take the discussion below for Michigan where we have data on violent crimes by permit holders. During the 2015-16 year, permit holders might have accounted for 0.053% of violent crime in the state. They claim that Michigan’s violent crime rates rose by 8.8% after their right-to-carry law was adopted, that is 166 times larger than permit holder’s share of violent crimes. To put it differently, for Donohue, Aneja, and Weber’s results to be plausible, police departments would have to be missing 99.4% of cases where permit holders have committed violent crimes. For other states the numbers below show similar results: Louisiana police would have to miss 99.5% of crimes committed by permit holders, Oklahoma would have to miss 99.93%, Tennessee 99.98%, and Texas 99.54%.
These percentages assume that no crimes are stopped or deterred by permit holders. To the extent that is true, these percentages would have to be even larger.
In any case, even our numbers overestimate any crimes that might arise from permitted concealed handguns. There are two reasons for this. First, virtually none of even these few violent crimes by permit holders were committed with guns. Second, when you are talking about 600,000 concealed handgun permit holders in Michigan at least a few of them would have committed violent crimes even if there wasn’t a right-to-carry law in the state.
Louisiana: Here are the percentages of permit holders who were charged or convicted of any type of felony, whether violent or nonviolent (aggravated assault is one type of felony, but felonies also typically include traffic violations). Including charged cases skews the number substantially, since permit holders have very low conviction rates in general. After all, permit holders are usually arrested even if they used their guns in justifiable self-defense. Police and prosecutors can’t just let them off the hook until they are sure about what happened. The vast majority of these cases are unlikely to involve firearms, however. (reports)
Share of violent crimes
2015: There were 25,208 violent crimes in Louisiana, with 19 felony charges or convictions against permit holders. Assuming that these felonies were all violent and that the accused were guilty of the charges, permit holders would account for just 0.08% of the total. Again, this is an overestimate of permit holders’ share of violent crimes.
2014: There were 23,983 violent crimes in Louisiana, with 15 felony charges or convictions against permit holders. Assuming that these felonies were all violent and that the accused were guilty of the charges, permit holders would account for just 0.06% of the total. Again, this is an overestimate of permit holders’ share of violent crimes.
The authors claim that the violent crime rate in Louisiana rose by 15.4% after the law was their right-to-carry law was adopted. Even if we use all felonies committed by permit holders to estimate their share of violent crimes, for their results to hold, police departments would have to be missing 99.5% of cases where permit holders have committed a violent crime.
Michigan: Below is the percentage of permit holders who were convicted of aggravated assault (with and without a weapon)
Also, the percentage of permit holders who were convicted of any type of violent crime (Murder, manslaughter, criminal sexual conduct, armed robbery, unarmed robbery, aggravated assault)
These 22 cases compare to a total of 41,231 violent crimes in Michigan, that is a 0.053% share.
These 18 cases compare to a total of 42,348 violent crimes in Michigan, that is a 0.044% share.
Again, for their results to be plausible, police departments would have to be missing 99.4% of cases where permit holders have committed violent crimes.
Minnesota: Permit revocations due to any type of assault
There were 7,094 aggravated assaults in Minnesota in 2015 and no concealed handgun permit holders were convicted of these crimes. There were also no revocations for other violent crimes. The authors here claimed a -0.7% drop in violent crime after the law was passed.
Oregon: Permit holders who were convicted of any type of felony, violent or nonviolent. The vast majority of these cases are unlikely to involve firearms.
There were 10,468 violent crimes in Oklahoma in 2015. Even though felonies involve more violent crimes, the 19 felonies that permit holders were convicted of in 2016 equal only 0.182% of violent crimes.
The authors claim that the violent crime rate in Oregon fell by -0.6% after the law was their right-to-carry law was adopted.
Oklahoma: Permit holders who were convicted of any type of felony. (reports)
There were 16,506 violent crimes in Oklahoma in 2015. Even though felonies involve more violent crimes, the 16 felonies that permit holders were convicted of equal only 0.097% of violent crimes.
The authors claim that the violent crime rate in Oklahoma rose by 9.7% after the law was their right-to-carry law was adopted. Even if we use all felonies committed by permit holders to estimate their share of violent crimes, for their results to hold, police departments would have to be missing 99.93% of cases where permit holders have committed a violent crime.
Pennsylvania: We were able to obtain data on revocations by 12 counties for aggravated assaults over the five years from 2012 to 2016. Those counties had an annual rate of 0.0004% of permits revoked for aggravated assaults with a weapon. It is hard to reconcile this tiny number with their claim that violent crimes rose by 26.5% in that state.
Tennessee: Revocations due to any type of assault other than vehicular assault
Permit holders who were convicted of any type of felony.
There were 40,400 violent crimes in Tennessee in 2015. Even though felonies involve more violent crimes, the 31 felonies that permit holders were convicted of equal only 0.077% of violent crimes.
The authors claim that the violent crime rate in Tennessee rose by 29.5% after the law was their right-to-carry law was adopted. Even if we use all felonies committed by permit holders to estimate their share of violent crimes, for their results to hold, police departments would have to be missing 99.98% of cases where permit holders have committed a violent crime.
Texas: Convictions for aggravated assault with any type of weapon
There were 67,727 aggravated assault in Texas in 2015. Even though felonies involve more violent crimes, the 31 felonies that permit holders were convicted of equal only 0.077% of violent crimes.
The authors claim that the violent crime rate in Texas rose by 16.6% after the law was their right-to-carry law was adopted. For their results to hold here, police departments would have to be missing 99.54% of cases where permit holders have committed a violent crime.
Utah: Convictions for aggravated assaults with and without a weapon (permits)
2016: 3 convictions for aggravated assault. 687,382 Permits. Percent of permit holders who are convicted of an aggravated assault: 0.00044%
2015: 4 convictions for aggravated assault. 632.276 Permits. Percent of permit holders who are convicted of an aggravated assault: 0.00063%
2014: 4 convictions for aggravated assault. 590,118 Permits. Percent of permit holders who are convicted of an aggravated assault: 0.00068 (all revocations 273 — rate is 0.046%)
2013: 1 convictions for aggravated assault. 535,857 Permits. Percent of permit holders who are convicted of an aggravated assault: 0.00019 (all revocations 320 — rate is 0.0597%)
There were 4,046 aggravated assault in Utah in 2015. Permit holders accounted for only 0.099% of aggravated assaults, though this probably exaggerates their share because two-thirds of Utah’s permit holders live outside of Utah where their aggravated assaults likely occurred. Their paper claims that Utah’s violent crime rate fell by 20.2%.
II. “individuals who carry guns around are a constant source of arming criminals”
Besides a few anecdotal stories, the paper points to the 600,000 cars that were stolen in 2013 out of about 250 million cars that were on the road (about 0.2%). The simplest solution to this problem is to eliminate gun-free zones that limit places where people can carry their guns. In addition, people with valuable items in their cars might be more likely to park their cars in secure areas so that the share of car thefts that occur with valuable items such as guns will undoubtedly be lower than the 0.2% rate for all cars.
But there is a more important problem with this argument. It is the belief that there aren’t many close substitute ways for criminals to obtain guns. For example, even if you could cause all guns to disappear from the US, how long do you think that it would take drug gangs to bring them into the country along with the drugs that they smuggle in? Probably almost immediately. Just as drug gangs can bring in drugs they can bring in the guns that they need to protect that very valuable property. One needs only to look Mexico where there are no legally owned guns by civilians for drug gangs to steal. Mexican drug gangs aren’t using 22 caliber handguns to commit their crimes.
III. “as more citizens carry guns, more criminals will find it increasingly beneficial to carry guns and use them more quickly and more violently to thwart any potential armed resistance.”
The evidence indicates that when civilians are carrying guns, gun crimes fall relative to overall crime. See this paper by Olson and Maltz from the Journal of Law and Economics as well as Lott’s book “More Guns, Less Crime.” So why this paper might speculate on what relationship might be, it is contrary to the empirical evidence.
IV. “Fourth, the passage of RTC laws normalizes the practice of carrying guns in a way that may enable criminals to carry guns more readily without prompting a challenge, while making it harder for the police to know who is and who is not allowed to possess guns in public. Having a “designated permit holder” along to take possession of the guns when confronted by police seems to be an attractive benefit for criminal elements acting in concert (Fernandez et al., 2015; Luthern, 2015). Fifth, it almost certainly adds to the burden of a police force to have to deal with armed citizens. A policemen trying to give a traffic ticket has far more to fear if the driver is armed.”
The question is what do police think about this. Do they think that concealed carry reduces the efficiency of law enforcement? Clearly, the answer is “no.” PoliceOne, the largest private organization of police in the United States with 450,000 members (380,000 active duty full-time police and 70,000 retired), asked whether legally armed citizens helped reduce crime.
“Do you support the concealed carry of firearms by civilians who have not been convicted of a felony and/or not been deemed psychologically/medically incapable?” 91.3% say “Yes, without question and without further restrictions.”
“Considering the particulars of recent tragedies like Newtown and Aurora, what level of impact do you think a legally-armed citizen could have made?” 86.2% of the officers said that “Casualties would likely have been reduced” or “avoided altogether.”
“On a scale of one to five — one being low and five being high — how important do you think legally-armed citizens are to reducing crime rates overall?” 76.4% of officers said that legally-armed citizens were either very important or extremely important in reducing crime.
Anyone who has talked to police officers knows that police are actually very relieved when they find out that a permit holder is driving the car that they are pulling over because they know how law-abiding permit holders are. The Castile case in Minnesota is the only case of its type.
V. “All of these factors are a tax on police, and therefore one would expect law enforcement to be less effective on the margin, thereby contributing to crime.”
As just noted, police strongly disagree with this claim and it would seem that if permit holders made the job of police more difficult, they would have noticed.
VI. Murder rate results are completely misleading.
Throughout their paper, Donohue, Aneja, and Weber use murder data in which well over 70 percent of the county-level sample has zero murders. Including these zero values without using the appropriate statistical tests creates widely understood biasing effect, which in this instance has the effect of giving the appearance that right-to-carry laws increase crime. While they do use negative binomial regressions in some of their estimates, they never discuss this problem, many others have discussed it at length. Of the 22 estimates that they provide using negative binomials, there are three statistically significant results out of 22 estimates. Two of the three significant results are just from where they look at data over only the 2000 to 2014 period. The problem here is that they don’t seem to understand that their results here are proving the opposite of what they claim for the simple reason that the states that adopted right-to-carry laws over the 2000 to 2014 period had much more restrictive permitting rules and issued permits during those years at a slower rate than states that adopted concealed handgun laws before 2000.
Since the authors never discuss the problem of zero values and the implications that it has for many of their results, here are a few quotes from past work. From More Guns, Less Crime (University of Chicago Press, 2nd edition in 2000 and 3rd edition 2010):
p. 285: The “truncation problem,” which occurs in county-level data sets because in some years many counties do not experience certain types of crimes at all—80 percent have no murders, for instance. If the murder rate in a county is zero before the law goes into effect, simple randomness means that sometimes the crime rate will go up, but the reverse cannot happen— crime rates cannot fall below zero. This could bias results for these regressions toward finding an increase in crime from the law. To avoid that, they exclude counties where there were no crimes committed.
p. 288: . . . [Duggan] ignoring the “truncation problem,” noted above by Plassmann and Tideman, and thus treating counties with no reported murders the same as the others. For example, when most counties have zero murders in any given year, no matter how good the law is, murder rates can’t fall any further. But simple randomness can mean that sometimes you will see the crime rate rise from zero even though it had no connection with the right-to-carry laws. In his last set of estimates, his analysis of the different violent crime categories included counties with zero crimes. There are a number of ways to statistically adjust for this problem (Tobit, negative binomials, etc.), but Duggan didn’t bother to use these techniques—thus biasing his results against finding a drop in crime.
Chapter 2 of MGLC shows that statistics for all violent crime rate categories suffer from this problem to some extent, though it is primarily seen for murder and to a lesser extent for rape and robbery. Plassmann and Tideman suggest a way of solving this problem in the 2001 Journal of Law and Economics (p. 772):
These analyses ignore the fact that crime rates cannot fall below zero. We argue below that this practice makes their results unreliable for crimes with low crime rates, and we suggest that a count analysis is more appropriate. While the standard approach to estimating such count-data models is to undertake a maximum likelihood analysis using the Poisson or the negative binomial distribution, we find that a Markov chain Monte Carlo analysis of a Poisson-lognormal model is easier to implement and yields more precise estimates for the crime data.
See also Plassmann and Whitley (Stanford Law Review, 2003):
A major problem with county-level data is what one should do with all the observations that have zero crime rates. Including arrest rates creates the problem of eliminating observations whenever a countyís crime rate is zero. This occurs because the arrest rate is defined as the number of arrests divided by the crime rate, and it is not possible to divide by zero. On the other hand, with weighted least squares, omitting the arrest rate is not a useful suggestion either. Including all counties with zero crime rates will bias the estimated benefit of the concealed handgun law towards finding an increase in crime, because no matter how good the law is, it cannot lower the crime rate below zero. Although the crime rate cannot fall in those counties, there will be some occasions, even due to pure randomness, where the crime rate rises. This problem occurs with Ayres and Donohueís table 1 estimates. . . .
VII. Why looking at average crime rates before and after the law are not a useful way of measuring impact of crime rates
Despite all these problems with their estimates, only one of their 12 “spline” estimates in their first 8 tables show a statistically significant increase in crime rates after right-to-carry laws are adopted. They find 7 significant results in their twelve estimates that look at simple before and after averages (what they call their “Dummy variable model”), but one had hoped that academics would have long ago realized that such estimates are not very reliable. Here a discussion from the second edition of “More Guns, Less Crime” (2000, pp. 214-5).
VIII. Faulty assumption that all Right-to-carry laws are the same and the impact of ignoring the fact that more recently adopted right-to-carry laws have had higher fees and longer training requires.
Here is an abstract from a 2014 paper. The point here is equally applicable to the period up through 2014 that was studied by Donohue, Aneja, and Weber:
Unfortunately, many who have examined the impact of so-called “shall-issue” or “right-to-carry” laws assume that the adoption of such laws causes a large, immediate increase in the number of permits. But that is often not the case, for states differ widely as to how easily permits can be obtained. This problem is particularly problematic for studies that have looked at the period after 2000. In fact, the share of the adult population with permits increased less during the 1999-2010 period in the states that adopted right-to-carry laws than the states that they are being compared against.
IX. Synthetic approach
In the synthetic approach, that Donohue, Aneja, and Weber use they use from 2 to 4 states to predict changes in crime rates for right-to-carry states. But to say that it is arbitrary what states that they end up using for comparisons is an understatement. For example, if you wanted to predict how Georgia’s violent crime rate would change over time, would your first choice be California and New York? If you wanted to predict the changing violent crime rates in states as diverse as Colorado, Idaho, Minnesota, Mississippi, Nebraska, and Utah, would you primarily or almost exclusively rely on Hawaii? Similarly, Louisiana and Tennessee might be surprised that these authors think that you should rely on Chicago (where half of Illinois’ violent crimes occur and the vast majority of changes in Illinois’ violent crime rate arise).
To put it mildly, their results are crucially dependent on what states that they pick to compare, and they have a lot of control over which states that they pick. The issue isn’t really whether they use rules that result in Hawaii or some other state to make the comparison, but the fact that how they pick the states and the number of states that they use to compare is arbitrary.
X. Lumping all violent crimes together
Other papers in this areas have looked at all the different types of violent crime (such as murder, rape, robbery and aggravated assaults), so it is quite unusual to only examine them as a group and it hides how other crime rates are changing. Let’s assume for a second that aggravated assaults. Crimes involving aggravated assaults are on average much less harmful to people than the other violent crimes: murder, rape and robbery. It is simply not honest to lump all these crimes together as it would hide the true changing cost of violent crime.
There are many other points that have been previously raised and could be raised again, but these are three major mistakes in the Donohue, Aneja, and Weber paper. But the media doesn’t even try to get both sides of the story, eschewing the most fundamental tenet of journalism. The authors also commit intellectual malpractice by refusing to address longstanding objections to their methods, which create the false impression that right-to-carry laws increase violent crime.
Points II, III, IV, and V have been added in the update of this report.
A response to Donohue’s new response is available here.