For the third year running, the Centers for Disease Control and Prevention describes its annual estimate of firearm injuries as “unstable,” which means it is potentially misleading and unfit for use by researchers. The Consumer Product Safety Commission, which operates the database that serves as the basis for those estimates, is now taking the first step toward a possible solution: Bringing in independent reviewers to look at the faulty data and provide recommendations on how to fix it.
The assessment will scrutinize the sample of hospitals that feeds the CPSC’s nonfatal injury database. As The Trace and FiveThirtyEight first reported, that sample is currently very small, which can result in wildly unreliable estimates. According to an excerpt of the work contract provided by a CPSC spokesperson, the review will seek to “understand the pros and cons of keeping the current samples, expanding the current samples, or resampling the hospital emergency departments.” The review will also suggest new methods for making calculations for certain types of injuries that vary widely between hospitals. Our reporting has revealed that shootings are one such type of injury.
The CDC’s newly released figures for gun injuries in 2018, as well as those from the two years prior, are so problematic that they have been suppressed on the agency’s public data portal. According to the last figures published before the CDC instituted this change, anywhere between 31,000 and 236,000 people were injured by guns in 2017. The hidden estimates leave a vacuum in a source that researchers rely heavily on: We previously identified more than 50 academic papers that had cited the CDC’s gun injury estimates since 2010.
The CPSC assessment is being conducted by Westat, a company that specializes in health, education, and policy research and which holds many federal contracts. The agency has asked Westat to revise the hospital selection procedure for the database twice in the past, first in 1988 and again in 1996, which is the last time the sample design was modified. Both revisions addressed “changes in the universe of hospitals,” refining the database to better reflect hospital openings and closures since its creation. The contract for the new assessment was awarded for $116,000. It is expected to be completed in late September, at which time “CPSC staff will review the assessment and work with the federal partners funding [the database to] determine next steps,” the spokesperson said.
Guohua Li, editor-in-chief of the medical journal Injury Epidemiology and founding director of Columbia University’s Center for Injury Science and Prevention, said in an email that the assessment is an appropriate step for the CPSC to take. He added that Westat is “well qualified to take on the project.”
The CPSC database currently uses a sample of roughly 60 hospitals to generate estimates of nonfatal injuries. Last year, The Trace and FiveThirtyEight found that the replacement of a single hospital in the sample caused the national firearm injury estimate to swing dramatically, increasing the 2015 figure by more than 22,000. According to the CPSC contract, Westat will also “document alternative methods for replacing hospitals that drop off the samples.”
After we reported on the flawed CDC estimates, in October 2018, a group of U.S. senators led by Democrat Bob Menendez of New Jersey sent a letter to the head of the Department of Health and Human Services, urging him to fix the data. The CDC’s director, Robert Redfield, responded by pledging to “improve the precision and accuracy of [its] nonfatal firearm injury estimates.”
Li said that expanding the hospital sample would be a “practical and an effective approach to addressing the data issue.”
Because the existing CDC estimates are so uncertain, it’s not clear just how wrong they are. But the advocacy group Everytown for Gun Safety published an independent estimate of nonfatal gun injuries last year with figures that are considerably lower — and more reliable — than the corresponding CDC estimates for the same years. (Everytown provides grant funding to The Trace.) Everytown’s analysis was based on detailed data from another federal database which is available to researchers for a fee. The analysis found an estimated 62,856 injuries in 2013; 79,127 in 2014; and 78,006 in 2016. The corresponding CDC figures based on the CPSC hospital sample are 84,258; 81,034; and 116,414, respectively.
While the Everytown report is a useful stopgap, it’s not a solution to the problem of bad data. As FiveThirtyEight’s Maggie Koerth wrote, “nonprofits can’t replace the federal government on gun violence research.”
