For journalists, researchers and the general public, the Centers for Disease Control and Prevention serves as an authoritative source of information about Americans’ health, including estimates of how many people are killed or injured by guns. The agency’s most recent figures include a worrying uptick: Between 2015 and 2016, the number of Americans nonfatally injured by a firearm jumped by 37 percent, rising from about 85,000 to more than 116,000. It was the largest single-year increase recorded in more than 15 years.
But the gun injury estimate is one of several categories of CDC data flagged with an asterisk indicating that, according to the agency’s own standards, it should be treated as “unstable and potentially unreliable.” In fact, the agency’s 2016 estimate of gun injuries is more uncertain than nearly every other type of injury it tracks. Even its estimates of BB gun injuries are more reliable than its calculations for the number of Americans wounded by actual guns.
An analysis performed by FiveThirtyEight and The Trace found that the CDC’s report of a steady increase in nonfatal gun injuries is out of step with a downward trend we found using data from multiple independent public health and criminal justice databases. That casts doubt on the CDC’s figures and the narrative suggested by the way those numbers have changed over time.
In response to a detailed list of questions and an analysis memo showing that there may be issues with its gun injury data, a CDC spokesperson said in an email that the scientists involved in gathering and analyzing the data “are confident that the sampling and estimation methods are appropriate.”
To produce its estimates of gun injuries, the CDC uses data collected by the Consumer Product Safety Commission. We asked the commission about the single-year jump of 31,000 gun injuries, and a spokesperson replied, “Although visually, the [CDC] estimates for firearm-related assaults appear to be increasing from 2015 to 2016, there is not a statistically significant difference between the estimates.”
Over a dozen public health researchers reviewed The Trace and FiveThirtyEight’s analysis and said that the inaccuracy of the CDC gun injury data has serious implications for the national-level understanding of gun violence.
“No one should trust the CDC’s nonfatal firearm injury point estimates,” said David Hemenway, director of the Harvard Injury Control Research Center.
But many researchers have trusted these numbers, or at least referenced them. Since 2010, at least 50 academic articles have cited the CDC’s gun injury estimates. Last year, for example, the American Journal of Epidemiology published a paper that used CDC data to conclude that there was a “hidden epidemic of firearm injury.”
“For those of us who are doing this kind of research, it’s disconcerting,” said Priscillia Hunt, a researcher at the RAND Corporation, a nonprofit research organization. “With the CDC, there’s this general assumption that they are reliable and have good data.”
Hunt herself cited the estimates in the introduction of a policy paper she published last year.
“It feels like a gotcha moment for people who’ve used it,” she said.
One likely reason that researchers frequently cite the data is that it’s easily accessible online through the CDC’s Web-based Injury Statistics Query and Reporting System, or WISQARS. Measures of the estimates’ uncertainty are also available through the system, but they are hidden by default.
Kristen Moore, a professor of mathematics at the University of Michigan, recently cited the CDC’s gun injury estimates in an article for an actuarial trade publication. She was surprised to see that trends from other sources of data pointed in the opposite direction from the trend in the CDC’s data.
“WISQARS is so nice and you can go there and do your own queries and you can pull the data yourself, and it has the name CDC behind it, which makes it seem so official,” she said.
The Trace and FiveThirtyEight compared estimates for 10 types of injuries, including gun injuries, using data from the CDC and two other data sets produced by the Agency for Healthcare Research and Quality, a branch of the Department of Health and Human Services.
In order to account for the differing methodologies used to produce the estimates, we compared which way the data trended in each data set, rather than the estimated tallies themselves.
If the various data sets are each accurately measuring the same underlying phenomenon, the trends should be similar; a large mismatch in the direction or slope of trend lines would indicate that one of the data sources may be using a flawed approach to data collection or estimation.
Among seven types of injury, the data sets reflected similar trends. Three types of injury — gunshots, drowning and cuts — showed statistically significant differences between the CDC and at least one of the other data sets, with the CDC diverging most dramatically from the other two when it came to gun injuries.
The FBI and Bureau of Justice Statistics also collect data that give us another way of evaluating the CDC’s estimates. The FBI tracks gun crimes, though not injuries specifically, and the BJS tracks gun injuries by surveying crime victims. Neither of these databases include other sources of gun injuries, such as accidents or suicide attempts — but they offer another frame of comparison for the CDC’s estimates. These data sets show an uptick in gun injuries and gun crime from 2015 to 2016 — which you can see by looking at the two rightmost dots on each panel of the charts below — which is similar to what the CDC shows. But here too, the multiyear trends — represented by the lines on the charts — diverge from what the CDC shows.
After seeing the comparisons between the CDC’s data and other data sets, Linda DeGutis, a former director of the National Center for Injury Prevention and Control at the CDC, questioned whether the agency’s model for estimating the number of gunshot injuries nationwide is reliable.
“I don’t know when the last time was when someone took a look at the methodology,” she said. “A federal agency should be able to compare the data sets that are available to say, ‘Wait a minute, why are we seeing these discrepancies?’”
A spokesperson from the Consumer Product Safety Commission said in an email that the sample’s design “requires a more complex calculation to detect changes over time” than the type of analysis that The Trace and FiveThirtyEight performed. The information to do the more complex calculation, however, is not available through WISQARS.
The data source the CDC uses to generate nonfatal injury estimates may be uniquely unfit to track gunshot wounds. Unlike the agency’s fatal injury counts, which are tallied from death certificates by the CDC itself, its estimates of nonfatal injuries are derived from a survey of hospitals, known as the National Electronic Injury Surveillance System, or NEISS. The database, which is maintained by the Consumer Product Safety Commission, was originally created with the objective of tracking consumer-product-related injuries. Injury information is collected from a sample of hospitals that is designed to act as a cross-section of all the nation’s hospitals: large and small, urban and rural.
The NEISS has several shortcomings that directly limit its ability to accurately quantify gun injuries. The most significant is its size: The database collects records from a sample of approximately 100 hospitals in the U.S. that may vary slightly from year to year as hospitals leave the sample and are replaced. Of these 100 hospitals, 66 — less than 2 percent of all hospitals in the United States — are used to collect additional data that includes gunshot wounds. (For comparison, the emergency room injury database produced by the Agency for Healthcare Research and Quality surveys nearly 1,000 hospitals.)
What’s more, the hospital selection process fails to take into account regional variations in injuries. For many common types of injuries, such as burns and falls, this doesn’t present a problem. But some injuries, such as gunshot wounds, vary strongly by location; they tend to be intensely concentrated in certain areas and nearly absent from others.
Consider two hospitals, one in a low-crime area of Manhattan and the other in an area of Chicago where gun violence is common. Both may treat a similar number of patients, but they’ll probably see vastly different numbers of gunshot victims. This variation in shooting injury volumes is a problem for statistical modeling. Because there are so few hospitals in the sample, the presence or absence of a single facility can dramatically affect the estimate. If the Manhattan hospital goes into the model, the estimate will skew lower. If the Chicago hospital is used, the estimate will be higher.
These geographic variations likely contribute to the high degree of uncertainty in the CDC’s gun injury estimates. The CDC assesses this uncertainty using a measure called the coefficient of variation, which calculates the relative amount of error associated with an estimate. Higher values indicate larger potential errors and less reliability.
The coefficient of variation for the CDC’s most recent firearm injury estimate is 30.6 percent — higher than every type of injury except drowning/submersion. (Estimates of drowning-related injuries are also vulnerable to this geography problem — one hospital near a body of water, for example, could distort the totals.) By comparison, the average coefficient of variation for gun injury estimates from the Nationwide Emergency Department Sample, a similar database that tracks visits to emergency rooms, is less than 8 percent.
Another indication of the uncertainty in the CDC’s estimates is the confidence interval, which is essentially the high and low ends of an estimated range of injuries, where the real number, whatever it is, likely falls somewhere in between. In the case of the CDC’s 2016 gun injury estimate, the confidence interval indicates there is a 95 percent chance that the true number of total gun injuries falls somewhere between a low of 46,524 and a high of 186,304.
“Basically, the confidence intervals are enormous,” said Hemenway, the Harvard Injury Control Research Center director. “So you have no idea about trends.”
The Consumer Product Safety Commission did not respond to direct questions on whether its data was suitable for generating reliable estimates of the national number of gun injuries.
The consensus among the researchers interviewed by The Trace and FiveThirtyEight is that the rise in the CDC’s gun injury estimate is likely caused by some hospitals in the sample treating more gun injuries than others. “If you have hospitals in communities that have a lot of firearm injuries, it’s going to change the estimates in a very significant way,” said Guohua Li, editor-in-chief of the peer-reviewed medical journal Injury Epidemiology and director of Columbia University’s Center for Injury Epidemiology and Prevention.
To check this geographic hypothesis, The Trace and FiveThirtyEight filed Freedom of Information Act requests for the lists of hospitals included in the NEISS sample from 2000 to 2018. Both the CDC and the Consumer Product Safety Commission initially declined our request, the CDC on the grounds that it does not gather that data, and the consumer commission on the grounds that releasing the hospitals’ names would be a violation of patient privacy. We appealed the commission’s decision; the appeal was granted and we have been told that we’ll receive the full list of hospitals soon. We will analyze the data once we have it and follow up with any significant findings.
Investigating America’s gun violence crisis
Reader donations help power our non-profit reporting.
For now, external research supports our suspicions. A 2017 study co-authored by Phil Cook at Duke University and published in the American Journal of Public Health established how heavily hospital selection can skew injury estimates in the NEISS. Cook said the research found that the addition of just five gun-treatment hot spots to the pool of hospitals in the CDC data accounted for much of the apparent increase in nonfatal gunshot injuries from assaults. When Cook adjusted his model to account for the distortion from these hot spots, the trend in nonfatal shootings from year to year appeared to be flat.
“NEISS is used to estimate all kinds of injuries and not just gunshot injuries, so when the Consumer Product Safety Commission looks at their sample, they’re not necessarily concerned about what is going on with any one of those [injury types],” he said. “But the CDC should be.”
Other databases have grappled with similar geographic distortions. In 2014, for example, federal researchers at the Agency for Healthcare Research and Quality set out to reduce the effects of geography and hospital selection in the National Inpatient Sample, a database of hospital admissions. Instead of looking at all cases from a subset of hospitals, as they had done in the past, researchers decided they should look a subset of cases from all participating hospitals. By expanding the number of hospitals included in the sample — from 1,000 to more than 4,500 — the redesigned database nearly halved its margins of error. Because the database now draws on data from most of the hospitals in the country, its estimates are no longer so susceptible to the distortions that can result from a small sample size.
A study published last year by the Journal of the American Medical Association found that compared to other leading causes of death, there has been a disproportionate lack of research on gun violence. Another article in JAMA’s journal of internal medicine indicated that language in the 1996 congressional appropriations bill likely contributed to a lack of funding for gun research — that act includes a provision known as the Dickey Amendment that says CDC money may not be used to “advocate or promote gun control.” Former leaders in the CDC have said that the agency made a conscious choice to avoid firearm research rather risk political retribution.
“We have a lost generation of firearms research,” said Sandro Galea, dean of the Boston University School of Public Health. Galea, who contributed to the “Hidden Epidemic of Firearm Injury” article that uses NEISS data, says there are limitations to every available data set. “We piece [together] what we know from various other sources. There is no one comprehensive approach.”
Researchers have proposed a number of potential solutions for a more reliable national estimate of nonfatal gunshot injuries. Daniel Webster, director of the Johns Hopkins Center for Gun Policy and Research, suggests expanding the FBI crime-tracking program to also track gun injuries, since many police departments are already recording this information. Kit Delgado, a professor of emergency medicine and epidemiology at the University of Pennsylvania, thinks researchers should interrogate the NEISS data and decide what the database can contribute that can’t be provided by newer systems like those maintained by the Agency for Healthcare Research and Quality. Every researcher we interviewed agreed that adding more hospitals to the CDC’s data would improve its reliability.
Expanding the CDC’s National Violent Death Reporting System to track nonfatal injuries is at the top of Li’s list of suggestions. He says that most of the infrastructure is already in place, and ultimately he believes that “it is more important to know why those victims of firearm injuries survived than what happened to those who died.”
In his work as editor-in-chief of Injury Epidemiology, Li has recently seen more manuscripts citing the Agency for Healthcare Research and Quality’s data instead of the CDC’s.
“Given that the [AHRQ] data are based on much larger samples, it provides more reliable national estimates,” Li said. “There is no compelling reason for researchers to use the NEISS data in the future to assess the time trends.”
Clarification: This article has been updated to clarify that the databases used for comparison against the CDC data are produced by a branch of the Department of Health and Human Services, the Agency for Healthcare Research and Quality.
