Donate

Problems with Siegel et al, American Journal of Public Health, “Easiness of Legal Access to Concealed Firearm Permits and Homicide Rates in the United States”

Mar 8, 2018 | Featured

Problems with Public Health Research

Siegel, Xuan, Ross, Galea, Kalesan, Fleegler, and Goss have a paper in the American Journal of Public Health that claims to be the first paper to find a statistically significant increase in homicides from the passage of right-to-carry laws.

The bottom line is that even accepting the regressions that the Siegel at all ran, they didn’t  calculate their levels of statistical significance properly. and that if that is done, their results are not statistically significant.

Professor Carl Moody and Dr. John Lott wrote this letter to the AJPH.

Dear Letters Editor:

Even accepting Siegel et al’s paper on its own terms, it misstates the statistical significance of its results and excludes critical data.1 The authors mischaracterize the existing literature and ignore critical issues raised by previous researchers.

Siegel et al’s estimates have significant serial correlation. This leads to a large difference between the cluster-robust and the default standard errors that the authors report.  After correcting for this flaw, their “main” result for total homicides is no longer statistically significant at the 5 percent level for a two-tailed test.  Clustering the standard errors reduces the t-statistic from 3.65 (p=.00) to 1.67 (p=.095). (The 95 percent confidence interval is [.991, 1.12].)

The authors also throw out observations for states that no longer require a permit for concealed carry.  This is particularly strange given that the paper indicates that more liberal concealed carry regulations increase homicides.  If this were true, these states should be particularly dangerous.  But when those observations are included in the paper, the dummy variable for those laws is actually negative (though not statistically significant).  Just as importantly, including this dummy variable makes it so that the shall-issue dummy is no longer statistically significant at the 10 percent level for a two-tailed t-test.

Siegel et al cut falsely claim: “In 2 studies, shall-issue laws were found to decrease homicide rates. In 2 studies, these laws were found to increase homicide rates.”

But 8 other published papers use state-level data and find that concealed carry laws reduce homicide (other studies find this benefit looking at county or city-level data).2-9  And, in fact, neither of the papers that they cite to show that concealed carry laws increase murder actually find the increase they claim.

Zimmerman’s paper provides two sets of estimates.10 While the first set does show statistically significant results, Zimmerman warns, “However, as the shall-issue law impact is being identified from only eight state changes in the data, it is difficult to give any strong causal interpretation to these estimates.”  When he provides his preferred instrumental estimates, which adjust for certain biases, Zimmerman finds no statistically significant results.

As to Ludwig’s results, they were never statistically significant.11  In discussing Table 4, he writes: “The standard errors imply that the point estimate is not statistically significant.”  None of the results in Table 5 are statistically significant at the 10 percent level for a two-tailed test.

There are numerous other problems with this paper because it ignores concerns raised in the previous literature.  For example, simple dummy variables for laws can be quite misleading.  If crime rates are rising before a law is adopted and falling afterwards, depending on the relative rates of those pre- and post-law changes, dummy variables could be positive, negative or zero even though the laws clearly produced a statistically significant change in trends.  And no attempt is made to account for the huge differences in shall-issue laws. In Illinois, it costs at least $450 to get a permit. In Pennsylvania, it is $20.

It is a mistake to focus on homicides and not on murder, which does not include justifiable homicides (most jurisdictions in the US don’t collect data on justifiable homicides, and even those that do underreport it).  Shall-issue laws may lead to an increase in justifiable homicides, but Siegel et al would interpret that as a bad result.  Still, the difference between homicides and murders in 2015 was equal to 3,363 deaths (about 26% of the homicides).

Siegel et al is the only published paper to actually claim that right-to-carry laws increase homicides or murder, and even they don’t really find such a result as they miscalculated their levels of statistical significance and dropped out data.

Sincerely,

John R. Lott, Jr., Ph.D.
Crime Prevention Research Center
Carlisle E. Moody.
Department of Economics
College of William & Mary

  1. Siegel M, Xuan Z, Ross CS, Galea S, Kalesan B, Fleegler E, Goss K. Easiness of Legal Access to Concealed Firearm Permits and Homicide Rates in the United States. AJPH. 2017:107:61-67.

  2. Mustard, D. The Impact of Gun Laws on Police Deaths. J Law Econ. 2001;44: 635-657.

  3. Lott JR JR, Whitley JE. Safe-Storage Gun Laws: Accidental Deaths, Suicides, and Crime. J Law Econ. 2001;44: 659-689.

  4. Olson D, Maltz MD. Right-to-Carry Concealed Weapon Laws and Homicide in Large U.S. Counties: The Effect on Weapon Types, Victim Characteristics, and Victim-Offender Relationships. J Law Econ. 2001;44: 747-770.

  5. Lott JR JR, Whitley JE. Abortion and Crime: Unwanted children and out-of-wedlock births. Econ Inquiry. 2006: 45: 304-324.

  6. Moody CE, Marvell TB. On the Choice of Control Variables in the Crime Equation. Oxford Bulletin of Econ & Stat. 2010: 72: 696–715.

  7. Moody CE, Marvell TB, Zimmerman P, Alemante F. The Debate on Shall-Issue Laws. Rev Econ & Fin 2014: 10:71-81

  8. Giusa M. An examination of the effects of concealed weapons laws and assault weapons bans on state-level murder rates. Applied Economics Letters. 2014: 21,4: 265-267.

  9. Barati M. New evidence on the impact of concealed carry weapon laws on crime. Int Rev Law Econ. 2016;47: 76-83.

  10. Zimmerman PR. The deterrence of crime through private security efforts: theory and evidence. Int Rev Law Econ. 2014;37:66–75.

  11. Ludwig J. Concealed-gun-carrying laws and violent crime: evidence from state panel data. Int Rev Law Econ. 1998;18:239–254.

Response letter from the AJPH

Dear Dr John Lott,

Thank you for your letter which I carefully examined and compared with the information we have on file about this paper. It appears that all the points have been addressed in some ways during the review process or in the paper:

1. Serial correlation: the paper says “We used robust standard errors that account for the clustering of observations, serial autocorrelation, and heteroskedasticity. 23”
2. 4 states that no longer require a permit for concealed: these states were in the original submissions. The removal was requested by reviewers and the editor because their number was too small. Their removal did not affect the overall interpretation of the results.
3. Data at the county or city level: These references were removed from the original submission because the paper is about state level data.
4. Homicides vs murders: The last line of Table 3 only includes “Supplementary homicide report (OMB form no. 1110–0002), offense 1a. Murder and nonnegligent manslaughter. Washington, DC: Federal Bureau of Investigation; 2017. Available at: https://ucr.fbi.gov/nibrs/addendum-for-submitting-cargo-theft-data/shr. Accessed June 17, 2017.

I therefore don’t think that there is a need of a letter to clarify what is actually in the paper or what was considered by peer reviewers inappropriate to be in the paper. If you propose an alternative analysis of the available data, it needs to be described in detail, submitted as an article, and I will be happy to have it should undergo a review process as rigorous as that undergone by Siegel et al’s paper.

Sincerely,

Alfredo Morabia, MD, PhD
Editor-in-Chief, AJPH

 

Our response to the editor.

Dear Alfredo:
Thanks for your email on Friday.  Regarding the discussion that you quote from their paper on robust standard errors their footnote 23 cites White (1980).  While White (1980) standard errors will correct for heterskedasticity, it does not correct for serial correlation because the procedure does not cluster the standard errors. In Stata, to get standard errors clustered at the state level (to correct for serial correlation), you must invoke the option “vce(cluster state)” or “robust cluster(state)”. If you invoke the option “robust”, you get heteroskedastic consistent standard errors, but not consistent with respect to serial correlation.
Since the authors used STATA, here is even a discussion from the STATA handbook that makes clear that it may be necessary to “then use cluster-robust errors to mop up the remaining and/or introduced serial correlation” (https://www.stata.com/meeting/13uk/nichols_crse.pdf).  Attached are our log file, do file, and data file from the our replication of Siegel et al.’s results.  (Please also note that negative binomial regressions are not appropriate for these estimates because they deal with count data, and the data here are continuous.)  You can then see how clustering knocks down the estimated significance so that it is just barely significant at the 10% level.
The authors provide no test for the level of serial correlation that persists after they have used robust standard errors.  A test for serial correlation clearly in the log file shows that there is positive serial correlation and it is statistically significant with an F-test of 13.08, which is statistically significant at the 0.0007 level.
 
Possibly Siegel et al aren’t familiar with the types of adjustments that they were making and didn’t read the paper by White or possibly they didn’t read the STATA manual to find the limits to the adjustment that they made.  But whatever the explanation, a quick look at the attached STATA log file will show you that doing this does affect their results. 
 
As to the second point, you can see from the attached files that with the removal of the data that they claim “did not affect the overall interpretation of the results” they are no longer statistically significant at even the 10% level.
 
As to the third point, just concentrating on the state level data, the two papers that they claim previously showed a bad effect from concealed handgun laws did not (indeed, attached below is an email from Siegel where he conceded as much to me and they miss 8 other published papers that use state-level data and find that concealed handgun carry laws reduce homicide.  Thus, rather than saying that previous research was divided on whether these laws increased homicide, none had previously shown it and the large majority had shown the opposite.
 
As to point four, neither any of their results nor their data file contain data on murders.  Homicides include justifiable self-defense and if you are looking at permitted concealed handguns, including justifiable homicides would cause a positive bias in any results.
 
All the necessary information is attached for you to replicate the test for serial correlation and the impact that actually correcting for it has on the statistical significance of their results.  Or the impact of them excluding data from their sample.  Please ask someone who deals with statistics whether robust standard errors can correct for serial correlation or whether you need to also do clustering.
 
Thank you for your time.  We look forward to hearing back from you.
Regards,
John
The final response from the Journal.
Dear John,

I appreciate your comments and disagreement with Siegel et al’s paper. But it is not something that can be addressed in letters. It is too substantial and complex. Noone will follow. As I said I will be happy to receive your alternative analysis as a research paper. It does not need to be a response to Siegel et al. Simply write up your own, stand alone paper, presenting what you think is the correct approach to the problem, your findings, and your conclusions. Having two peer-reviewed papers leading to opposite conclusions on this issue would constitute a solid basis for a discussion and a clarification.

Hopefully you will be convinced.

Sincerely,

Alfredo Morabia, MD, PhD
Editor-in-Chief, AJPH

johnrlott

1 Comment

Archives