The Merit of Minimizing Mistakes
Progress across biomedicine and health is attended by a series of well-knit statistical inferences.
Statistical inference tells you whether that obscure human trait that has seized your imagination is apt to be a blind alley or a useful (and publishable) biomarker of disease risk. It estimates how many willing patients you’ll need to test your new drug or device. It reveals a pathogen’s infection rate and whether this or that public health measure is apt to be effective. It tells you which genes are differentially expressed between different conditions and which patient subgroups are apt to respond best to which personalized treatments.
In the fiscal year concluded in June 2023, the curiosity, ingenuity and confidence of Vanderbilt University Medical Center biomedical scientists was met with $532 million in federal and nonfederal research support, including grants and contracts. As a research hub, VUMC is acrawl with statistical design, bias detection and analysis.
As new biomedical research findings come to light, the role of biostatisticians may go unnoticed and unsung, but their contributions aren’t lost on their many research partners, whose decisions and discoveries tend to hinge on data collection and analyses led by biostatisticians.
Yu Shyr, PhD, the Harold L. Moses Professor of Cancer Research and professor and chair of Biostatistics at VUMC, was asked by his Taiwanese mother to describe his work in a nutshell. “I said, ‘Mom, I deal with variation, I deal with uncertainty.’”
Uncertainty: the statistician’s bread and butter.
Biomedical research teams must wade a tide of uncertainty arising from the variability and complexity of biological systems and the limitations of the measuring apparatus.
Arriving at Vanderbilt in 1994 as the sole biostatistician in the cancer center, Shyr found that biostatisticians were often regarded by other faculty members as simply technicians.
“There’s been a great cultural change. VUMC investigators now view us as true partners, as collaborators,” he said.
The growth of biostatistics at VUMC coincided with investments at the national level in clinical, translational and epidemiological research and the rise of genomics and other omics data. Becoming a free-standing department of the Vanderbilt University School of Medicine in 2003, the Department of Biostatistics is today “the mainstay of team science throughout the Medical Center, throughout the university,” said VUMC Chief Scientific and Strategy Officer Jennifer Pietenpol, PhD, speaking in October 2023 at a symposium marking the department’s 20th anniversary.
“Over the last 15 or 20 years, I would argue that Vanderbilt Biostatistics revolutionized pragmatic clinical trials,” said Wesley Self, MD, MPH, Senior Vice President for Clinical Research and another speaker at the same event. Self was lauding the contributions of biostatisticians to VUMC trials in which patient-by-patient randomization was tactically abandoned in favor of randomizing clusters of patients.
Traditional randomized clinical trials require the recruitment of hundreds, if not thousands, of patients into studies that compare potential treatments against inactive placebo. These studies can take months, if not years, not to mention millions of dollars, to complete. Patients in the intensive care unit can’t wait, Self said during the symposium. The acuity of their illness also makes it difficult to enroll them into traditional clinical trials.
Eight years ago, Self and his colleagues set out to settle a long-standing medical conundrum: Which is the better intravenous fluid to give patients on admission to the ICU — regular saline, which contains salt, or balanced crystalloids, which more closely resemble the liquid part of blood? Intravenous saline, which has been used in medicine for more than a century, has been linked to an increased risk for acute kidney injury and death. However, prior studies had failed to prove that the risk could be lowered by giving balanced crystalloids instead.
“The problem with studying something so routinely used in clinical care is that … you can get an IV fluid bag within seconds of admission to the ICU,” Self said. “People are getting fluids too rapidly for us to randomize them.”
So, Vanderbilt biostatisticians Li Wang, MS, and Dan Byrne, MS, helped design a pragmatic, cluster-randomized, multiple-crossover trial, in which five ICUs at VUMC were randomly assigned to use saline during even-numbered months and balanced crystalloids during odd-numbered months.
The trial, which enrolled nearly 16,000 patients over the course of 23 months, found that patients in the balanced-crystalloid group had a significantly lower incidence of adverse kidney events or death within 30 days of admission compared to the saline group.
Then came COVID-19.
As the pandemic swept into the United States, the Foundation for the National Institutes of Health set up a public-private partnership called Accelerating COVID-19 Therapeutic Interventions and Vaccines (ACTIV) to evaluate potential treatments and vaccines, streamline clinical trials, and expedite rapid approval and production.
In early 2021, the Vanderbilt Institute for Clinical and Translational Research was selected to lead a national trial of treatments targeting the renin angiotensin aldosterone system in patients hospitalized with COVID 19 under the ACTIV 4 Host Tissue platform. Comprised of three hormones, RAAS is essential for the regulation of blood pressure and fluid balance. SARS CoV 2, the virus that causes COVID 19, can disrupt this system, resulting in life-threatening complications, including fluid in the lungs, blood clots and severe cardiac injury.
“This is a brand-new world, right?” said Self, who helped lead the ACTIV-4 RAAS Clinical Coordinating Center at VUMC. The urgency of the situation required a different approach to clinical trials. “We need to know, and we need to know very quickly, what drugs might help people survive. We want rapid, efficient and rigorous clinical trials maximizing knowledge generation with lowest possible sample size.”
Biostatistics again came to the rescue. Biostatistician Matt Shotwell, PhD, and colleagues in the ACTIV-4 RAAS Data Coordinating Center decided to use a new outcome measure for judging the effectiveness of potential therapies. Rather than mortality, the historical endpoint for many clinical trials, they chose oxygen-free days, which measures the time hospitalized patients with acute lung injury require supplemental oxygen. It enabled them to detect signs of recovery earlier, and in smaller groups of patients.
Among the first drugs to be evaluated were two that were thought to help the body recover from a pathological imbalance caused when SARS-CoV-2 binds to and interferes with ACE2, a vital enzyme which controls blood pressure and blood flow to multiple organs. The study, conducted at 35 sites in the United States between July 2021 and April 2022, found that neither drug was effective for treating severe COVID-19.
While the findings were disappointing, they also were important, Self said at the time. “We now understand that giving these drugs, which would seem very logical from a mechanistic perspective, should not be done,” he said.
In 2023, VUMC was chosen to lead a national effort to better understand acute respiratory distress syndrome (ARDS), pneumonia and sepsis, which together kill hundreds of thousands of people in the United States each year.
Previous studies have been stymied by heterogeneity. Pneumonia currently is defined by symptoms such as fever, cough and shortness of breath. But the same cluster of symptoms may have entirely different root causes in different patients. That’s why a treatment aimed at a broadly defined disease may benefit only a percentage of patients. “We need to identify mechanisms of disease,” Self said, “and target therapies at those underlying mechanisms to advance clinical care.”
Thus, the ARDS, Pneumonia, and Sepsis Phenotyping Consortium was born. VUMC biostatisticians Frank Harrell Jr., PhD, and Bryan Blette, PhD, help lead the consortium’s Design, Data, and Statistical Analysis Unit, which will oversee collection and analysis of laboratory tests and imaging studies from thousands of patients nationwide.
The goal is to link symptoms and other observable characteristics to specific genetic or physiological mechanisms, and thus define the three syndromes more precisely.
Biostatisticians ought to be prepared to deliver unwelcomed conclusions to their research partners, says Harrell, professor of Biostatistics, who founded the Department of Biostatistics as its first chair.
“We’re maybe the most collaborative of all the sciences. We love working with our clinicians, our basic researchers, translational researchers. And we like it when they smile,” Harrell said. “If you have to tell a researcher that because the signal-to-noise ratio in their data is not great, and they don’t have enough patients to make up for that, and it’s best to move to a different hypothesis, you don’t really get a lot of smiles. But that might be the very best message that a statistician could deliver.”
VUMC is a “data heaven” that attracts top researchers, Harrell said. “The quality of the clinical researchers and biomedical researchers at Vanderbilt is a recipe for success for biostatistics. If you look at the quality of these collaborators, the statisticians have a dream situation.”
Harrell, speaking from his deep experience, offers bracing observations about what’s often wanting in academic biomedical research and the resulting published science. For example, he finds that studies often wallow in omics-type prognostic novelties without bothering to measure the advantages of these variables, if any, over everyday models used in clinics the world over. (About omics, Harrell said, “What’s happened over the years is the research has gotten better. It’s gotten to be more reliable; the measurement platforms are more accurate. But the findings that are meaningful are not coming in large numbers.”)
It’s not uncommon, Harrell notes, for scientists to gloss over alternate explanations of their study findings, sometimes with disastrous consequences. “You have papers retracted. You have papers criticized online, sometimes very viciously. You have grants that might not renew because somebody tried to replicate your work, and they couldn’t do it.
“If you’re rigorous as you go, you’ll try to prevent big mistakes, trying to keep your mistakes small.”
Biostatistics-informed research project design keeps costs down and wins grant decisions, Shyr says. Biostatisticians should in any event be involved at the start before data collection begins. “The PI wants to ask their question in the most efficient way. As a partner, the biostatistician can help you work through that question,” he said.
For the best hope of directly changing clinical practice in the near term, research needs to deal in variables routinely available in the electronic health record, “but this question is often overlooked,” said Dandan Liu, PhD, associate professor of Biostatistics.
The coordination process varies when starting work on a research project with other investigators, Liu said. Having long collaborated with researchers in the Vanderbilt Memory and Alzheimer’s Center and the Department of Emergency Medicine, she said that where there is already familiarity with the dataset or investigators, fewer meetings are required, and much of the communication can be managed through email.
“If it’s something that is less developed, and I need to help in making suggestions about how you might consider reformatting or reorganizing your aim, the initial meeting will take a bit longer. But most likely, it requires one meeting for me to understand the context and what preliminary data are available, and then we can just move forward with emails,” said Liu, director of the methods program at the Vanderbilt Institute for Clinical and Translational Research and director of the newly established Vanderbilt Biostatistics Data Coordinating Center.
Outside of training program support from VUMC’s central administration, biostatistics is largely funded by grants brought in by researchers across the institution from the National Institutes of Health and other public sources. Shyr said his tenured faculty, while continuing to work alongside researchers from other departments, are increasingly acting as principal investigators, or PIs, on their own grants.
Bryan Shepherd, PhD, professor of Biostatistics, works as a PI on methods for validating electronic health record data for use in research. His work on projects led by other investigators often involves HIV and global health. These studies typically involve Shepherd and a staff biostatistician meeting with the PI every few weeks.
“It’s actually pretty exciting. It’s fun to be involved. There’s a lot of meeting with people, learning about what they’re doing, trying to help them out,” he said. “And the longer-term projects often bring up interesting statistical problems that can feed into the sort of things that I do where I write my own grants.
“My favorite part of my job is coming up with new ways of analyzing data, working with students and other faculty in that process, and then applying it to real data and showing that our new method works better than the current standard.”
In long-term collaborations, Harrell said, “You’re collaborating before the next question is formalized, and thus you have a lot greater chance, with a lot of help from the biostatistician, of formulating a problem that’s answerable.” That’s in contrast to the assistance offered through daily biostatistics clinics, to which the department devotes considerable resources.
“People come in who we don’t know,” Harrell said, “and they have just started to work on something and maybe they’ve started collecting some data, which is actually to their disadvantage. It’s not uncommon that we have to say during one of our clinics to an investigator, ‘That question you have is a really interesting clinical question or public health question, but we don’t think it’s going to be feasible with the kind of data you have.’”
And then there are the myriad basic science labs at VUMC. Here, Harrell notes, almost anything can be randomized, and measurements can be far better controlled than those taken in the clinic.
Do these researchers have any need of biostatisticians?
“They need biostatisticians to a monumental degree,” Harrell said.
“I totally agree,” Shyr said. “Any research involving experiment design will benefit from discussion with a biostatistician. Translational researchers, epidemiological researchers, investigators examining biomarkers — all those people know how to work with biostatisticians. Pure basic science researchers, they may not see the benefit yet. I think we still have room to improve in that respect.”
At VUMC and elsewhere, biostatistics, the most collaborative of all sciences, depends on willing research partners.
– by Paul Govern and Bill Snyder