Fact Checker Snopes.com’s big mistakes in comparing mass public shootings in the US and Europe

11 Mar , 2018  

Snopes.com recently fact-checked a post that we first put up in June 2015 and updated on January 7, 2016.  This became an issue for them because a story that Fox News’ Special Report that had run a couple years ago was getting circulated on Facebook after the Florida high school shooting.  They wrote: “Our conclusion is that this is accurate based on the CPRC’s definition of a mass shooting, but also extremely misleading. It uses inappropriate statistical methods to obscure the reality that mass shootings are very rare in most countries, so that when they do happen they have an outsized statistical effect.”

They made two general points: questioning our definition of mass public shootings and that we obscure how rare these attacks are in European countries.


Definition of mass public shooting.

We used the traditional FBI definition of mass public shootings in all our posts on this (e.g., here, here, and here).  There are several parts to this definition.

  1. The official FBI definition of mass public shootings excludes “shootings that resulted from gang or drug violence” or that occurred in the commission of another crime such as robbery.
  2. The FBI also includes only shootings in “public places” such as: commercial areas (malls, stores and other businesses); schools and colleges; open spaces; government properties (including military bases and civilian offices); houses of worship; and healthcare facilities.
  3. From the 1980s to 2013, the original FBI definition of “mass killings” had been “four or more victims slain, in one event, in one location,” and the offender is not included in the victim count (CRS, July 30, 2015).  In 2013, the definition was changed to “three or more killings.”  Vast majority of academics have continued to use the four or more definition.  This includes researchers such as James Alan Fox.  See also studies years ago such as Grant Duwe, Tom Kovandzic, and Carl Moody, “The Impact of Right-to-Carry Concealed Firearm Laws on Mass Public Shootings,” Homicide Studies, Nov. 1, 2002.  Even groups such as Bloomberg’s Everytown, which Snopes cites approvingly, have recently used the four or more definition. The other organization that Snopes cites approvingly, Mother Jones, also has used the four or more definition for most of the period and has only recently well after we did this report started using the three or more definition.

The claim that the CPRC Study “obscure[s] the reality that mass shootings are very rare in most countries”

Snopes.com claims:

The first thing to note about the rankings is that Lott has compared the mass shooting death rate in the United States with that of other countries where there was a mass shooting between 2009 and 2015. This might seem obvious, but it’s important to point out that very many countries did not see a single mass shooting as defined by Lott during this period.

The second striking thing about the list of mass shootings in Europe is that it is dominated by outliers. Where the United States saw at least twelve mass shooting deaths every year between 2009 and 2015, some of the other countries on Lott’s list experienced one or two rare but very high-casualty shootings. When you average out the death rates, this creates a highly misleading impression about the consistency and lethality of mass shootings outside the United States.

Looking at individual countries in Europe is a lot like examing individual states in the United States.  Snopes.com thinks one should look at the median yearly rate that mass public shootings occur.  If you want to compare median yearly rates that there are deaths in different countries in Europe, why not compare individual countries in Europe to individual states in the US.  The countries in Europe that had a mass public shooting had an average population of 23.5 million.  The total US was 323.3 million.

We can break down the US into individual states just as Europe is broken down into individual countries (click on the table below to enlarge). Snopes makes much of the fact that while Europe as a whole has mass public shootings every year, most European countries didn’t. Indeed, eleven of the fifteen European countries shown had only one year from 2009-2015 where they had a mass public shooting (if you include Russia, it would be eleven out of sixteen countries). But the same pattern holds true for US states, where 14 of the 18 states that had a mass public shooting experienced only one year with such deaths from 2009-2015. To put it differently, 73% of the European countries had deaths from a mass public shooting in only one year (69%, including the European part of Russia). By contrast, 78% of the states with a mass public shooting had deaths in only one year.

While it’s true that all European countries have a median mass public shooting death toll of zero, it is also zero for all but one of the 50 US states — California. Looking at average deaths among states and European countries, we find that six of the ten worst are European countries.

Snopes.com makes it look like our analysis only compared the US to individual countries in Europe.  They completely ignore that we also compared the US to the EU and Europe as a whole.  For four of these seven years, the total number of deaths from mass public shootings is greater in these European countries than in the US.  If one compares the yearly median for Europe to the yearly median for the US, they are virtually the same.  With or without France, the median for Europe is 19.  For the US, it is 18.

Take Snopes.com comments on our ranking of Norway:

The example of Norway gives a good illustration of just how absurd this use of statistics is. In 2009 and 2010, according to Lott’s data, there were zero mass shooting deaths in Norway. In 2011, far-right extremist Anders Behring Breivik killed eight people in a series of bombings in Oslo, then shot dead 69 more in a massacre at a Labor Party summer camp on Utoya island.

In 2012, 2013, 2014 and 2015, there was not a single death from a mass shooting in Norway.

If we compare the number of mass shooting deaths each year to the population of Norway each year, using the same method we did for the United States, we get an average annual death rate of 2 per million, more than 20 times higher than the rate in the United States (0.09 per million), even though we know there were zero mass shooting deaths in six out of those seven years, in Norway.

If this strikes you as ridiculous, you’re on to something.

Norway has a population of about 5 million people.  Seven states in the US have populations between 4.4 million (Kentucky) and 5.7 million (Wisconsin).  The other states are Louisiana (4.7 million), Alabama (4.9 million), South Carolina (4.9 million), Colorado (5.5 million), and Minnesota (5.5 million).  Three of those states had no mass public shootings.  The other four each had one mass public shooting, just like Norway.  The big difference is that those seven states had a total of 33 deaths from mass public shootings, while Norway had 69.  Even if you look at the mass public shootings in all the ten states between 4 and 7 million, the total number of deaths from mass public shootings is 48, with a total population of 78.5 million.  Their total population is 15.5 times greater than Norway’s.  Does Snopes.com really want to argue that Norway should only be compared to the US as a whole?

The following tables show the breakdown for casualties.

Other points

Snopes.com claims: “there are instances where the difficulty of categorizing mass shootings becomes clear.”  They give a couple of examples:

CPRC did not include the 7 June 2013 killings of five people in Santa Monica, California. John Zawahri shot and killed his father and brother at home, set the house on fire, then shot and killed three more people at Santa Monica College.

Although Zawahri killed five people, only three were killed in public in the second part of a shooting spree that was only ended when police killed him outside the college’s library. This might appear to many observers to be a mass shooting in which the attacker killed five people, but it was left out of the CPRC analysis, apparently because two of the five fatal shootings took place in a private home. Conversely, the CPRC analysis did include a 20 February 2012 incident in which Jeon Soo Paek shot dead his two sisters and their two husbands at the spa they co-owned in Norcross, Georgia.

In contrast to the public dimension of the Santa Monica shooting, all the victims in Norcross were relatives of the perpetrator, and all were killed at their family-owned place of business, reportedly after a dispute over money. However, Paek shot dead four people at a location that wasn’t a private home and — apparently on that basis — those deaths are included in the CPRC analysis.

But both of these definitions fit the traditional FBI definition.  As noted above, the FBI defined it as “four or more victims slain, in one event, in one location.”  The fact that two of the Santa Monica killings occurred in a private home also goes against the FBI definition of a “public places.”  As Snopes.com notes, the Norcross killings did involve four or more people killed at one time in a public place.


Snopes.com ignores our comparison of either the EU or Europe to the US and their second table ignores our concern about adjusting for different country populations.  If they are going to emphasize that countries in Europe with just a fraction of the US population rarely have mass public shooting (either not adjusting for population differences and/or looking at the “median” yearly rate), the only fair comparison is to look at individual states in the US.  Looking at Snopes.com’s desired measure of yearly median deaths, comparing Europe to the US or individual European countries to individual US states produces virtually identical results.  The same is true for average annual death rates, even if one adjusts for the population.  One thing completely ignored in Snopes.com’s discussion is the very high rate of casualties in Europe compared to the US.

The only thing that is ridiculous is Snopes.com’s comparison of small individual countries to the United States.  There is no reason to expect a tiny country like Norway to have the same absolute frequency of attacks as the United States.  The types of yearly median comparisons that they want to make only make sense if you compare similarly sized populations (Europe to the US or individual European countries to individual US states).


Compare Snopes.com’s critique of our work with their pretty favorable discussion of Michael Bloomberg’s Everytown when they put out their number of school shootings.  While even a long list of left-wing fact checker have criticized these claims (see here and here), Snopes.com can’t even criticize them for not violating their own claimed rule and using shots near a school instead of at a school.


7 Responses

  1. Tim Walley says:

    Thank You. I don’t know how many times I get lectured on how “fill in the blank western european country” has so few mass public shootings compared to the US,
    and that we must adopt their “sensible gun laws” so our numbers will drop.

    Damn you Lichtenstein and your perfect record 🙂

  2. Rhett Emerson says:

    Thank you Dr. John Lott Jr. For telling and explaining the truth. I keep having discussions with people all spouting the untruth of snopes, everytown etc etc . … I have read your book “War on Guns”. It is an excellent resource.

  3. Michael Prieur says:

    Doesn’t everyone get their statistical data from Snopes on gun deaths and every other topic? Maybe you should ask them what their credentials are, Dr. Lott?

    They are, in their mind only, the final word on truth. NOT.

  4. Dave Burris says:

    In other words, Snopes.com is just another politically inspired purveyor of fake news.
    Everything they say now has to be taken with a grain of salt, or less.

    • Lorenzo says:

      Agreed. Is anyone picking up here on the PREPOSTEROUS!?! You can’t use two different standards of normalization and then put the data sets side-by-side and present them as if they are an apples:apples comparison. Gimmie a freakin BREAK!!! This speaks VOLUMES to the continuing lack of credibility among the anti-gun movement.

    • johnrlott says:

      Thanks, Miami, note that you haven’t responded to any of the specific points that I have raised. Despite multiple emails to Snopes, they have also not responded. I finally wrote up an op-ed piece, but we will see if they even respond to that, but I doubt it.
      Media Bias Fact Check is not a serious organization. They cite Alan Dershowitz as saying that I have gotten funding from the NRA, but if you actually listen to the interview, Dershowitz backs off the claim when challenged. Does Media Bias Fact Check acknowledge that? No. As to “Junk Science,” neither the person who they cite at Rutgers nor Dershowitz have ever published one peer-reviewed academic empirical study on crime. I have published over 100 peer-reviewed articles and dozens are on crime related issues. Also, most of the peer-reviewed research supports my work. https://crimeresearch.org/2014/11/do-right-to-carry-laws-reduce-violent-crime/

Leave a Reply

Your email address will not be published. Required fields are marked *