Fact Checker Snopes.com’s big mistakes in comparing mass public shootings in the US and Europe

Mar 11, 2018 | Media Bias

Snopes.com recently fact-checked a post that we first put up in June 2015 and updated on January 7, 2016.  This became an issue for them because a story that Fox News’ Special Report that had run a couple years ago was getting circulated on Facebook after the Florida high school shooting.  They wrote: “Our conclusion is that this is accurate based on the CPRC’s definition of a mass shooting, but also extremely misleading. It uses inappropriate statistical methods to obscure the reality that mass shootings are very rare in most countries, so that when they do happen they have an outsized statistical effect.”

They made two general points: questioning our definition of mass public shootings and that we obscure how rare these attacks are in European countries.

Definition of mass public shooting.

We used the traditional FBI definition of mass public shootings in all our posts on this (e.g., here, here, and here).  There are several parts to this definition.

  1. The official FBI definition of mass public shootings excludes “shootings that resulted from gang or drug violence” or that occurred in the commission of another crime such as robbery.
  2. The FBI also includes only shootings in “public places” such as: commercial areas (malls, stores and other businesses); schools and colleges; open spaces; government properties (including military bases and civilian offices); houses of worship; common areas at apartment buildings; and healthcare facilities.
  3. From 1980 to 2013, the original FBI definition of “mass killings” had been “four or more victims slain, in one event, in one location,” and the offender is not included in the victim count (CRS, July 30, 2015).  In 2013, the definition was changed to “three or more killings.”  Vast majority of academics have continued to use the four or more definition.  This includes researchers such as James Alan Fox.  See also studies years ago such as Grant Duwe, Tom Kovandzic, and Carl Moody, “The Impact of Right-to-Carry Concealed Firearm Laws on Mass Public Shootings,” Homicide Studies, Nov. 1, 2002.  Even groups such as Bloomberg’s Everytown, which Snopes cites approvingly, have recently used the four or more definition. The other organization that Snopes cites approvingly, Mother Jones, also has used the four or more definition for most of the period and has only recently well after we did this report started using the three or more definition.

The claim that the CPRC Study “obscure[s] the reality that mass shootings are very rare in most countries”

Snopes.com claims:

The first thing to note about the rankings is that Lott has compared the mass shooting death rate in the United States with that of other countries where there was a mass shooting between 2009 and 2015. This might seem obvious, but it’s important to point out that very many countries did not see a single mass shooting as defined by Lott during this period.

The second striking thing about the list of mass shootings in Europe is that it is dominated by outliers. Where the United States saw at least twelve mass shooting deaths every year between 2009 and 2015, some of the other countries on Lott’s list experienced one or two rare but very high-casualty shootings. When you average out the death rates, this creates a highly misleading impression about the consistency and lethality of mass shootings outside the United States.

Looking at individual countries in Europe is a lot like examing individual states in the United States.  Snopes.com thinks one should look at the median yearly rate that mass public shootings occur.  If you want to compare median yearly rates that there are deaths in different countries in Europe, why not compare individual countries in Europe to individual states in the US.  The countries in Europe that had a mass public shooting had an average population of 23.5 million.  The total US was 323.3 million.

We can break down the US into individual states just as Europe is broken down into individual countries (click on the table below to enlarge). Snopes makes much of the fact that while Europe as a whole has mass public shootings every year, most European countries didn’t. Indeed, eleven of the fifteen European countries shown had only one year from 2009-2015 where they had a mass public shooting (if you include Russia, it would be eleven out of sixteen countries). But the same pattern holds true for US states, where 14 of the 18 states that had a mass public shooting experienced only one year with such deaths from 2009-2015. To put it differently, 73% of the European countries had deaths from a mass public shooting in only one year (69%, including the European part of Russia). By contrast, 78% of the states with a mass public shooting had deaths in only one year.

While it’s true that all European countries have a median mass public shooting death toll of zero, it is also zero for all but one of the 50 US states — California. Looking at average deaths among states and European countries, we find that six of the ten worst are European countries.

Snopes.com makes it look like our analysis only compared the US to individual countries in Europe.  They completely ignore that we also compared the US to the EU and Europe as a whole.  For four of these seven years, the total number of deaths from mass public shootings is greater in these European countries than in the US.  If one compares the yearly median for Europe to the yearly median for the US, they are virtually the same.  With or without France, the median for Europe is 19.  For the US, it is 18.

Take Snopes.com comments on our ranking of Norway:

The example of Norway gives a good illustration of just how absurd this use of statistics is. In 2009 and 2010, according to Lott’s data, there were zero mass shooting deaths in Norway. In 2011, far-right extremist Anders Behring Breivik killed eight people in a series of bombings in Oslo, then shot dead 69 more in a massacre at a Labor Party summer camp on Utoya island.

In 2012, 2013, 2014 and 2015, there was not a single death from a mass shooting in Norway.

If we compare the number of mass shooting deaths each year to the population of Norway each year, using the same method we did for the United States, we get an average annual death rate of 2 per million, more than 20 times higher than the rate in the United States (0.09 per million), even though we know there were zero mass shooting deaths in six out of those seven years, in Norway.

If this strikes you as ridiculous, you’re on to something.

Norway has a population of about 5 million people.  Seven states in the US have populations between 4.4 million (Kentucky) and 5.7 million (Wisconsin).  The other states are Louisiana (4.7 million), Alabama (4.9 million), South Carolina (4.9 million), Colorado (5.5 million), and Minnesota (5.5 million).  Three of those states had no mass public shootings.  The other four each had one mass public shooting, just like Norway.  The big difference is that those seven states had a total of 33 deaths from mass public shootings, while Norway had 69.  Even if you look at the mass public shootings in all the ten states between 4 and 7 million, the total number of deaths from mass public shootings is 48, with a total population of 78.5 million.  Their total population is 15.5 times greater than Norway’s.  Does Snopes.com really want to argue that Norway should only be compared to the US as a whole?

The following tables show the breakdown for casualties.

Other points

Snopes.com claims: “there are instances where the difficulty of categorizing mass shootings becomes clear.”  They give a couple of examples:

CPRC did not include the 7 June 2013 killings of five people in Santa Monica, California. John Zawahri shot and killed his father and brother at home, set the house on fire, then shot and killed three more people at Santa Monica College.

Although Zawahri killed five people, only three were killed in public in the second part of a shooting spree that was only ended when police killed him outside the college’s library. This might appear to many observers to be a mass shooting in which the attacker killed five people, but it was left out of the CPRC analysis, apparently because two of the five fatal shootings took place in a private home. Conversely, the CPRC analysis did include a 20 February 2012 incident in which Jeon Soo Paek shot dead his two sisters and their two husbands at the spa they co-owned in Norcross, Georgia.

In contrast to the public dimension of the Santa Monica shooting, all the victims in Norcross were relatives of the perpetrator, and all were killed at their family-owned place of business, reportedly after a dispute over money. However, Paek shot dead four people at a location that wasn’t a private home and — apparently on that basis — those deaths are included in the CPRC analysis.

But both of these definitions fit the traditional FBI definition.  As noted above, the FBI defined it as “four or more victims slain, in one event, in one location.”  The fact that two of the Santa Monica killings occurred in a private home also goes against the FBI definition of a “public places.”  As Snopes.com notes, the Norcross killings did involve four or more people killed at one time in a public place.

Conclusion

Snopes.com ignores our comparison of either the EU or Europe to the US and their second table ignores our concern about adjusting for different country populations.  If they are going to emphasize that countries in Europe with just a fraction of the US population rarely have mass public shooting (either not adjusting for population differences and/or looking at the “median” yearly rate), the only fair comparison is to look at individual states in the US.  Looking at Snopes.com’s desired measure of yearly median deaths, comparing Europe to the US or individual European countries to individual US states produces virtually identical results.  The same is true for average annual death rates, even if one adjusts for the population.  One thing completely ignored in Snopes.com’s discussion is the very high rate of casualties in Europe compared to the US.

The only thing that is ridiculous is Snopes.com’s comparison of small individual countries to the United States.  There is no reason to expect a tiny country like Norway to have the same absolute frequency of attacks as the United States.  The types of yearly median comparisons that they want to make only make sense if you compare similarly sized populations (Europe to the US or individual European countries to individual US states).

Appendix

Compare Snopes.com’s critique of our work with their pretty favorable discussion of Michael Bloomberg’s Everytown when they put out their number of school shootings.  While even a long list of left-wing fact checker have criticized these claims (see here and here), Snopes.com can’t even criticize them for not violating their own claimed rule and using shots near a school instead of at a school.

johnrlott

23 Comments

  1. Tim Walley

    Thank You. I don’t know how many times I get lectured on how “fill in the blank western european country” has so few mass public shootings compared to the US,
    and that we must adopt their “sensible gun laws” so our numbers will drop.

    Damn you Lichtenstein and your perfect record 🙂

  2. Rhett Emerson

    Thank you Dr. John Lott Jr. For telling and explaining the truth. I keep having discussions with people all spouting the untruth of snopes, everytown etc etc . … I have read your book “War on Guns”. It is an excellent resource.

  3. Michael Prieur

    Doesn’t everyone get their statistical data from Snopes on gun deaths and every other topic? Maybe you should ask them what their credentials are, Dr. Lott?

    They are, in their mind only, the final word on truth. NOT.

  4. Dave Burris

    In other words, Snopes.com is just another politically inspired purveyor of fake news.
    Everything they say now has to be taken with a grain of salt, or less.

    • Lorenzo

      Agreed. Is anyone picking up here on the PREPOSTEROUS!?! You can’t use two different standards of normalization and then put the data sets side-by-side and present them as if they are an apples:apples comparison. Gimmie a freakin BREAK!!! This speaks VOLUMES to the continuing lack of credibility among the anti-gun movement.

    • johnrlott

      Thanks, Miami, note that you haven’t responded to any of the specific points that I have raised. Despite multiple emails to Snopes, they have also not responded. I finally wrote up an op-ed piece, but we will see if they even respond to that, but I doubt it.
      Media Bias Fact Check is not a serious organization. They cite Alan Dershowitz as saying that I have gotten funding from the NRA, but if you actually listen to the interview, Dershowitz backs off the claim when challenged. Does Media Bias Fact Check acknowledge that? No. As to “Junk Science,” neither the person who they cite at Rutgers nor Dershowitz have ever published one peer-reviewed academic empirical study on crime. I have published over 100 peer-reviewed articles and dozens are on crime related issues. Also, most of the peer-reviewed research supports my work. https://crimeresearch.org/2014/11/do-right-to-carry-laws-reduce-violent-crime/

    • Jason W

      Appeal to authority. The refuge of those too lazy to read 😉

    • Canof Sand

      Gee, a left-wing site claims right-wing sites are biased and left-wing sites aren’t. Whodathunkit!

  5. Simon

    I’ve had fun reading your articles.

    I have a couple of questions about your tables, if you don’t mind. (I’ll concentrate on the murders, rather than casualties, for briefness.)

    Shouldn’t there be a line “USA for population of states with attacks”, to compare with the line for Europe?
    I make it 320 deaths / 186 million / 7 years = 0.246.

    How did you calculate “Europe with total Europe population”?
    It looks like 333 deaths / 540 million / 7 years = 0.0881

    But where did 540 million come from?

    The EU has a population of 512 million, and it doesn’t include Norway, Switzerland, Macedonia, Serbia or Albania (with 5, 9, 2, 9, and 3 million populations, and 69, 8, 5, 19 and 4 deaths).

    So, an EU figure would be:
    (333 – (69 + 8 + 5 + 19 + 4)) / 512 / 7 = 0.064

    The continent of Europe (which does include those countries) has a population of 840 million, but that includes Russia’s 143 million.
    Take out Russia, and the value drops to roughly:
    333 / (840-143) / 7 = 0.068 (same as Texas)

    Take out Turkey, Azerbaijan, Georgia, Armenia and Cyprus as well, and you’re still around the 600 million mark, which would make the total roughly:

    333/(840-143-83-10-4-3-1)/7 = 0.080
    Between Texas and the USA as a whole.

    • Jason W

      I also noticed the first thing SIMON brings up though I see he pulled 320 million figure from the table based on his statement (that he would concentrate on murders).
      His second point has me more curious though. I’ve checked his math and it looks good. Where does the 540 figure come from?

      • Simon

        You’re right, I got the figure from the wrong table, it should have been:

        “USA for population of states with attacks” 199 Deaths / 186 million / 7 years = 0.153
        A much less significant difference from 0.135 (but still bigger).

        Another point about the tables; why exclude France?

        I expect it’s because the 2013 Paris, France attack was exceptional (coordinated and with a political motive) and arguably falls into the “involving struggles over sovereignty” class, since they were “fighting” for an Islamic State.

        But then, the population of France shouldn’t be excluded, because it also had other mass shootings. The correct figure for “Europe for population of countries with attacks” would then be: 203 deaths / 353 million / 7 = 0.082 (a little more than half the USA’s value).

      • Simon

        I’ve just noticed I made another mistake, sorry.

        The original article included two mass shootings in Russia (2012, 2013), as being part of Europe, so the population of Europe should probably be treated as the full 840 million, giving:

        Deaths by mass shootings in Europe per million: (333 + 6 + 6) / 840 / 7 = 0.059 vs. 0.153 in the US; not that much more than a third.

        (That’s with the French terrorist attack, without it, it’s 0.037.)

        “Europe for population of countries with attacks” would then be: 215 deaths / 496 million / 7 = 0.061, much less than half of 0.153.

  6. Olav Nilsen

    This is is interesting. It shows that even the most basic statistics are difficult to apply.

    When I first read the original CPRC study, my initial thoughts were much the same as Snopes’. Then I read this reply, and to some extent I do see the point regarding individual states in the US vs countries in Europe. However, the more I think about it the more I think it fails. It looks like an attempt to make the terrain fit the map, instead of the opposite.

    Why? Because it seems very arbritrary to think of Europe as one entity or country. Even within the EU (not to mention Europe as a whole) the differences are enormous. Culture, language, history, demographics, laws, economy, religion, military, wars, gun control and every other factor you might look at to explain mass shootings are extremely different from country to country. These are 100% autonomous nations, with their own government. It is not in any way an European version of the United States of America with its 50 states. Not even close. There is nothing Spain can do to lower crime rates in the UK, or UK can do to stop migration to Germany. There is something the US government can do to address problems within any of the 50 states in the USA – and thus it seems equally arbritrary to suddenly start treating these as independent countries. You might as well start adding Afghanistan, Iraq or Libya to the European pool of countries … it simply does not make any sense.

    Yes, I do realize that there are big differences between American states. Just like there are big differences between different regions within Spain, Belgium and Norway. But they, and the USA, are still all sovereign countries, with their own government, laws and at least some common identity and cultural or other traits that probably influence the number og mass shootings. Thus they have to be treated as seperate countries (or as one country when talking about the US), and this reply from the CPRC to the Snopes article is deeply flawed.

    PS I’m a Norwegian that ended up here after searching for views on our mass shooting in 2011, so please excuse my poor language.

    • johnrlott

      Thanks very much for your comments, Olav. We compare the US to Europe, compare the US to individual countries in Europe, and finally individual US states to European countries. Pick what comparison you want, but they all provide useful comparisons. As to comparing US states to European countries, it is true that we have a central US government, but we have much more variation in state laws than you within the vast majority of individual European countries. Some states ban so-called “assault weapons” or magazines above different sizes or require licensing and other regulations while other states do not. I suspect that there are greater demographic differences across US states then is true for European countries. I could go on, but the point is pick whatever comparison that you want to pick. All the results are roughly similar: Europe or individual countries in Europe have very high rates of mass public shootings and very high rates of casualties. I hope that this helps.

      • Olav Nilsen

        Well, the problem is that no matter which comparison I choose, it does not make any sense. The US vs individual European countries due to the same issues that Snopes pointed out (outliers and the use of the mean), the other two because of what I said in my previous post. I do know that US states have variation in state laws, but you still have your constitution. And there is obviously a common American “identity” (history, constitution, culture …),if you will , that very well could be the cause for mass shootings (i.e. it makes no sense to split the country up into states for comparison). That, you don’t see in Europe as a whole (i.e. it makes no sense to pool the countries together) albeit it might make some sense to pool together some regions of similar countries like Scandinavia or perhaps Be-Ne-Lux.

        The only thing I can see from the numbers is that USA has a problem, you can clearly see a pattern of this type of behaviour. In most European countries with mass shootings, you can’t see a pattern. When you can, it is due to muslim terrorism, which is more like a guerilla warfare – a well defined and explained threat. And it seems a bit off to include these in the analysis, but OK.

        So, USA has a problem and comparing them to European countries makes no sense since there simply isn’t enough data points to do any sorts of statistical analysis. Unless you pool the European countries together. Which you can’t, or at least shouldn’t in my opinion. I’m not even sure why you would compare them in any case. I can assure you that the mass shooting here in Norway (the only one in peace time) didn’t create a need for comparing ourselves to other nations. Hoping that it would be “normal” behaviour so that we wouldn’t have to do anything. Neither should any country that experiences such things do. No stone should be left unturned. Whether thar means to look into gun control or other parts of the society. But it has to be done with an open mind, not through deliberately poorly applied and interpreted statistics to prove preconceptions or to serve a political agenda.

        I guess we’ll never agree on this, and that’s OK. It was an interesting read anyway. Thanks.

        • Darren McLaughlin

          The E.U. does have a minimum standard for gun laws.

          “European Directive No. 91/477/EC sets minimum standards regarding civilian firearms acquisition and possession that EU Member States must implement into their national legal systems. The Member States are free to adopt more stringent rules, which leads to differences in the extent of legal access to firearms among EU countries.”
          https://en.wikipedia.org/wiki/Overview_of_gun_laws_by_nation#cite_note-234

          The U.S. is somewhat constrained by the Constitution, which is one reason gun laws vary by state. The point is that since the E.U. in fact does regulate guns for E.U. countries, both comparison to individual countries and to the E.U. as a whole are relevant, though I don’t believe comparison to non-E.U. “European” countries is. As has been mentioned, choose the comparison you think is most relevant. Also, what’s important are mass murders in general regardless of what the tool happens to be.

  7. Bill Cawthon

    The biggest problem with comparing the U.S. to European nations (or other countries) on the basis of the availability of firearms is the assumption that such access is the driving force in mass shootings. Is it a contributing factor? Perhaps, but it must be remembered that even by the lowest estimates of gun ownership, there are at least 80 million gun owners in the U.S. and the number is likely higher with some estimates placing it at more than 100 million.

    Looking at the motivations of mass shooters, we find that many of them are common experiences. This includes job losses, broken hearts, bullying, child custody disputes, a fascination with other mass shooters such as the Columbine killers and Adam Lanza and even a straightforward desire to kill people, as was the case with James Holmes. Sexual frustration was said to be the motivation of Elliot Rodger, who murdered six in Isla Vista, California (three by stabbing) and the vehicle attack by Alec Minassian in Toronto, Ontario that left ten dead.

    If guns were a significant factor in these incidents, we should expect the U.S. to have a much higher incidence of mass shootings. After all, with 300 million firearms in the hands of 80 million or more people, all of whom are subject to many of the same setbacks as those who do commit mass shootings or mass killings of any type, this should be the perfect recipe for far more attacks than we have seen.

    The difficulty with using mass shootings to justify new gun control measures is that none of the legislation yet proposed would have any significant effect on the pool of firearms available. Given the fact that mass killers will even commit murder to obtain firearms and the fact experience shows there would be widespread civil disobedience in the case of laws requiring registration of either owners or guns, these legislative proposals would not be able to live up to the promises made for them. So why pass laws that won’t work?

  8. Fitz Cavendish

    Almost all of these (non-politically motivated) acts are committed by people on prescription psychotropic drugs. I mean nearly 100%. The condition is called akathisia induced impulsivity and it’s the result of the liver’s inability to properly metabolize the prescription medication, resulting in mania. A test is available for this liver disorder, but the media covers this truth up to protect its advertising revenue from Big Pharma. The politicians keep a lid on it to keep their drug industry campaign contributions flowing. Anti-gun elements in both camps keep it quiet to further their agenda.

    This is the root cause of the problem. We had plenty of guns 100 years ago, but we did not have plenty of psychotropic medication. A nice compromise between the left and the right would be to force anyone taking this class of medication to turn in their guns. Otherwise, the right to bear arms should be the right to bear arms.

  9. Cauthon

    “Does Snopes.com really want to argue that Norway should only be compared to the US as a whole?” Of course they do. The facts are not on their side, so if they want to advocate for gun control laws, they have to resort to “ingeniously specious manipulation of the data”. IOW, they figure if they torture the data long enough it will confess. Thanks for the (OTOH) remarkably detailed and instructive analysis published here. I noticed in a map on Wikipedia that there is a huge area of the midwest and far western USA that has hardly any markers for mass shootings – from Kansas and Nebraska all the way up to Washington, but you have provided a lot more details for comparison with Europe. It is sad to see so many in California, in contrast with some comments you see online claiming that their anti-gun laws have protected them. There must be something we can learn from the frequency of these events in CA, and along the Boston – Washington corridor.

  10. Cauthon

    One more point: When they have to argue about which state to look at, or which country, all that sort of detail, it is clear that the facts are not going their way. As Bill Cawthon says above, if guns were causing the problem, we have lots of guns, we should all be dead. We had plenty of guns several decades ago and quite low crime rates. Through the 1960’s and on up to maybe the 1980’s we increased our reliance on gun control laws and crime increased. We should be looking for what variables correlate with the recent increase in mass murder, and I do not think we will find availability of guns fitting the pattern.

  11. Simon

    “So which country has the largest death rate from public mass shootings?

    Norway does. With an outlier mass shooting death rate of 1.888 per million, *mostly* due to the tragic shooting carried out by Anders Brevik in 2011, Norway is at the top of the list.” [emphasis added].

    It was uniquely due to the one and only mass shooting they have ever had in peace time. Not “mostly”, and that is completely unambiguous.

Trackbacks/Pingbacks

  1. Canada’s Worst Mass Killing in 30 Years Proves Lunatics Who Want to Kill People—Don’t Need Guns - Guns in the News - […] to a study by the Crime Prevention and Research Center — which was attacked by Snopes, who failed to prove it wrong —…

Archives