Flawed gun data lead to wrong conclusions, researchers say

Researchers and some media reports have suggested a so-called "medical triumph" is happening in American hospitals because homicide rates have decreased despite a rise in gun violence. But the idea didn't sound right to two gun researchers.

"What is bizarre is that you see these patterns being reported as if they are gospel, and it just doesn't square," said Philip Cook, a professor of public policy at the Sanford School of Public Policy at Duke University and lead researcher on a new study on the data.

Cook discussed his doubts about the phenomenon with Dr. Garen Wintemute, who knows firsthand how hospitals respond to gun violence. He works with these patients as an emergency medicine doctor at the University of California, Davis Medical Center and is known for giving more than $1 million to fund research on the medical treatment of gun violence.

Wintemute knew that hospitals were good at saving people who have been shot but that there haven't been many medical advances to change the numbers quite so much. He didn't believe the trend theory, either.

When the gun violence rate went up, the scholars knew, the homicide rate typically followed. They decided to check the numbers for themselves.

Their results were published Thursday in the American Journal of Public Health.

They put Ariadne Rivera-Aquirre on the case. Rivera-Aquirre, one of Cook's grad students at Duke, got the huge task of digging through the numbers used by the older gun studies.

The data most people use to understand trends in gun violence come from the National Center for Health Statistics and the National Electronic Injury Surveillance System All Injury Program. These surveys are not simple, straightforward counts. The government gathers data from a representative sample of hospitals and uses sophisticated calculations to come up with the best estimates.

Rivera-Aquirre's first research challenge was to figure out what hospitals were used in these samples.

"Ariadne was very good at persuading the sources to share it with us and look behind the normal reporting survey system," Cook said. "This was the big breakthrough, actually being able to get to the detailed data from the survey system, and we were very grateful to those researchers for sharing it."

The researchers say they found two main problems.

First, the hospitals used in this representative sample between 2003 and 2012 had changed over time, probably skewing the results.

Some of the hospitals that withdrew from the national sample were replaced with others that had a significantly larger number of gunshot cases, including assaults. So researchers were no longer comparing the same data from year to year, leaving the impression that gun violence was going up and homicides were going down.

The second problem is that the hospitals also seemed to be recording the cases a little better over time, marking fewer gun cases as having "unknown" causes.

So, the researchers concluded, "the case-fatality rate did not change, and trauma care improvements did not influence the firearms homicide trend."

Daniel Webster, a professor of health policy and management at the Johns Hopkins Bloomberg School of Public Health, where he serves as director of the center for gun policy and research, said the new study is a good one that points to a bigger issue.

The data that these surveys rely on aren't meant to capture only gun violence; they are more general and track all of the different kinds of cases hospitals treat. Webster, who was not involved in the new research, said the United States needs better gun-specific data.

"My biggest pet peeve, or concern, as it relates to data and gun violence is that we have very minimal information on nonfatal gunshot wounds, and our ability to track them becomes incredibly limited," he said.

There's no uniform national hospital-based system that records this information. The FBI's Uniform Crime Reporting system fails to distinguish between certain levels of gun violence, especially when someone is threatened with a gun but isn't wounded. And law enforcement officers who take initial crime reports may be inadvertently incentivized to report lower crime rates, he said. They may not always record that a gun was present in a robbery, for instance, if no one got hurt. That too downplays the real rate of gun violence.

"We don't have good standardized data across the board nationally," Webster said, and the study is a good reminder about how much more could be done. "It's really difficult to track down information about one of our most important public safety and public health problems facing the country, and we all hope that can change."