Department of Criminology and Criminal Justice Professors Peter Reuter and Greg Midgette, along with Jonathan Caulkins of Carnegie-Mellon University, advise that both researchers and the general public take the forthcoming findings of the 2020 National Survey on Drug Use and Health (NSDUH)—a nationwide, general population survey (GPS) of 70,000 Americans released each fall—with a grain of salt.
In their new paper published in Addiction, “Heroin use cannot be measured adequately with a general population survey,” Reuter, Caulkins and Midgette assess the accuracy of the NSDUH and discuss how the design of such GPSs can lead to significant underestimates of the use of expensive and addictive drugs in the United States—an underestimation of 75% or more of actual usage rates, where heroin is concerned.
Reuter, Caulkins and Midgette focused their research on the NSDUH as it is “among the best GPS” most often used to inform national drug policies and programs. Indeed, all government reports cite the NSDUH figures uncritically.
They studied heroin data specifically because “if this strong GPS fails at estimating the one best-known opioid, then a fortiori there are concerns with GPS-based estimates of problem opioid use more generally.”
“NSDUH is a survey that’s done every year, generally among people who live in households and who are over 12 years old, where they ask people about their health and risky behaviors, including the use of heroin,” explained Midgette. “But by asking people in households about the use of heroin, you drastically underestimate the amount of heroin that is being consumed and the number of people who are consuming heroin.”
The professors looked at four potential shortcomings of GPSs under normal circumstances—selective non-response, small sample size, sampling frame omissions and under-reporting. They then compared the results of the NSDUH to overdose mortality data, workplace drug testing data, the number of admissions to federally funded inpatient substance abuse treatment programs, and the Arrestee Drug Abuse Monitoring program (ADAM). ADAM collected data and urine samples from a sample of arrestees from 1987 to 2013, a population often excluded from GPSs.
Collectively, this data produced estimates of heroin use 16 times greater than the comparable estimate by the NSDUH. And this was long before the mental and emotional challenges caused by COVID-19 came into play.
“COVID made everything more complex,” said Midgette. “The nature of data collection about drug use behaviors is already a hard endeavor; it’s already hard to find out about risky behaviors that people are, for obvious reasons, not excited to talk to you about. And when we have already mismeasured and misplanned and the problem seems to be mounting and the drugs are becoming more dangerous—especially with regard to opioids, fentanyl is becoming an increasingly large share of what is in an opioid drug on the street—the disconnect is becoming larger between what we understand and what is actually occurring.”
The researchers’ solution to this burgeoning issue isn’t to toss out NSDUH altogether, but to change the way leaders view and position it.
“Often, the way that the Substance Abuse and Mental Health Services Administration [SAMHSA] reports drug and alcohol use data is ‘Here is the new release of NSDUH, and here is what the new NSDUH estimates are,’” explains Midgette. “If the practice was instead to say ‘Here’s what we know across all of these data sets that SAMHSA is responsible for,’ rather than annually relying on the release of one survey and having that be the gold standard, that will give us a better sense of where we have relatively strong information agreement and disagreement.”
“Our idea is the idea of triangulation,” Midgette concluded. “That is, taking a bunch of imperfect data together, recognizing their imperfections and the benefits they bring to the calculations, and using them to systematically come up with a better estimate.”
