May 6, 2021 7:00 AM

The Statistical Secrets of Covid-19 Vaccines

They’re really very good, and they’re the only way out of the pandemic. But a tour through the numbers could bring the vaccine-hesitant into the tent.

The creation and distribution of vaccines against Covid-19 has been as close as science gets to a miracle—the culmination of centuries of infectious disease research, a mastery of viral genomics, the creation of a whole new kind of vaccinology, and speed in the face of crisis. So it’s weird that even as the global pandemic worsens (infection and death rates have gone vertical in India and South America), the appetite for vaccination in the United States seems to have peaked. The seven-day average number of doses hit 3.38 million on April 13 and has been on the downslope ever since.

Some of that trend might be simple market saturation—lots of the people who were supposed to go first, like the elderly, have already gotten their shots. People with easier access to vaccination centers got them too, as did the ones who were able to manage the confusing internet-based ways of getting an appointment. Who’s left? People who are harder to reach and people who might be “vaccine hesitant.” They resist getting vaccinated because—well, actually, that’s confusing. Some might think the pandemic isn't their problem to solve. Others might have a misguided fear of vaccines generally. But one hypothesis is that the non-takers think their risk for severe illness is so low that it’s not worth risking minor (or very rare severe) side effects. Part of that reasoning might be because of the way the trials were designed—because of an original sin built into that vaccine miracle. And part of that might be because they misunderstand the statistics involved. So. Nerd out with me, won’t you?

Back in October I wrote that the design of vaccine trials was going to be a problem. Instead of pitting them head to head in one giant battle royale—hot vaccine-on-vaccine action, with standardized protocols—every company did its own. Each one collected slightly different information about slightly different groups of people. They were good studies! Three vaccines got emergency use authorizations from the US Food and Drug Administration; dozens more are still under research. But it’s very hard to compare them to each other. That, plus confusion over the statistical measures of how well vaccines work, might be providing quiet (and mostly incorrect) justification for some people’s hesitancy. All that adds up to a delay in the widespread inoculation that will set the world back on course.

Here’s to statistics—the cause of, and solution to, all of life’s problems.

When the new vaccines came out, their makers and the government touted their efficacy with some impressive numbers—95% for Pfizer, 94% for Moderna, 67% for J&J. Sweet!

But “efficacy” has a specific meaning in the world of vaccine statistics, and it’s not “Hey, if I get a shot, my chance of getting Covid is now just 5%!” Ha, no, you dope. Because your chance of getting Covid wasn’t 100% in the first place. See, vaccine efficacy is actually a relative risk reduction. It’s a ratio comparing the risk of infection in people who got vaccinated versus people who didn’t (the control group). Since the basic function of a vaccine is, indeed, to keep people from catching a disease, you can imagine that this number can end up being pretty big—regardless of those people’s chances of getting Covid.

You could also, of course, calculate the absolute risk reduction. That’s simply the difference in risk for someone in the treatment group versus someone in the control group. Here’s an example: Say you have 100 people who don’t get a vaccine, and you find that 10 of them catch the disease. So the baseline risk of getting it is 10%. And suppose that 100 other people get the vaccine, and only one of these gets sick. Their risk is 1%. The absolute risk reduction (ARR) is then just 9% (10% minus 1%), because the risk was already pretty low. But the relative risk reduction (RRR) is 90%—that reduction of 9% divided by the baseline risk of 10%.

As a commentary in Lancet Microbe pointed out last month, even with trials on tens of thousands of people, the absolute risk reductions in Covid-19 vaccine trials are teensy-tiny—a reduction in the risk of getting severe Covid of just 1.2% for Moderna and a scant 0.84% for Pfizer. “One of the main reasons why absolute risk reduction is not shown is because of the numbers. If you say, ‘It’s 95% effective’—wow!” says Piero Olliaro, an infectious disease researcher at the University of Oxford’s Centre for Tropical Medicine and Global Health and one of the authors of the Lancet Microbe article. “But if your absolute risk reduction is like 0.8% or whatever it was, so what?”

The key here, though, is that absolute risk reduction does change according to how at-risk the groups of people were in the first place. This pandemic has widely varying risks across populations, and those change over time. (For example, viral variants change how infectious Covid can be, and young people’s risk of severe illness and death has changed as social policies and infection rates have fluctuated. It’s a hard problem!) I’m suggesting that this confusion, and the conflation of these two ideas, might be at the heart of some hesitance. By not being clear about the different flavors of risk and benefit for different vaccines and different people, public health experts have let doubt and dodgy personal interpretations flourish.

Someone who’s hesitant to get a vaccination against Covid—not a full-on anti-vaxxer—might be worried about their own risks (of getting Covid-19 or vaccines), and unclear how they weigh against the benefit of almost certainly not getting Covid-19. Efficacy, or relative risk reduction, paints with too broad a brush—and brushes aside their personal assessments. “As individuals, we think of risk as ‘my individual risk.’ But the risk is a statistical calculation,” Olliaro says.

Absolute risk helps clarify the individual-risk part. It also helps with policymaking, because it allows people with calculators to figure out exactly how many lives they’re going to save. To really bring that into focus, the inverse of the absolute risk reduction—1/ARR, if you’re even fractionally interested—is called the number needed to vaccinate (NNV). Which is to say: How many people do you have to vaccinate to prevent just one case of Covid-19 (or one severe case, or one death—depending on the study’s endpoints)?

The different studies yield different results. As Olliaro calculates, it takes 76 people vaccinated with the Moderna two-dose regimen to prevent a single case and 117 for Pfizer’s nominally “hotter” and more sought-after two-dose vaccine … but just 84 for the one-dose J&J shot. But what does that tell you about the power of the vaccine? Maybe not so much, because the NNV also changes with the baseline risk of the population, which is affected by who they are and how prevalent infections are around them. (J&J conducted its trials, in part, in South Africa, where a more infectious variant of the virus was also spreading; that may have upped the risk for people there, and therefore lowered the NNV.)

All the approved vaccines are very, very good. But by one metric (efficacy), the J&J vaccine looks less good, and by another (NNV), it looks pretty hot now too. The same turns out to be true of the AstraZeneca vaccine, available in Europe. “Because of the higher risk in the unvaccinated group, they ended up having a better absolute risk reduction than the others, and the numbers needed to vaccinate are smaller,” Olliaro says.

This is an old question in the public health world—whether all these numbers help people or overwhelm them. But increasingly it looks like more information about vaccines alleviates hesitancy rather than exacerbating it. That comes down to how people see their risk profiles. “If you got Covid, you got Covid 100%, and if you don’t, it’s 0% Covid,” Olliaro says. “You have to consider the individual’s perspective within the community.”

One of the hallmarks of the pandemic has been that it affects different groups of people in different ways. In the US, poor people and people of color have been much more likely to get sick and die of Covid-19 than white people and rich people. Old people are at more risk than young people.

And like every other medical intervention ever, vaccines themselves have risks as well as benefits. The J&J and AstraZeneca vaccines have been associated with very rare but severe blood clots, which led to a pause in the use of the J&J vaccine in the US last month. People with severe allergies may be more likely to get anaphylactic shock from the mRNA-based two-dose vaccines.

All these complications create a fog around the decision-space, making some people’s risk-benefit calculations more complex—or creating a space for people who perceive themselves to be at low risk from Covid-19, or who are more concerned about side effects than they need to be, to think it’s OK to not get vaccinated. “Most people aren’t sitting there with numbers worrying about the decimal point, thinking, ‘I’m going to weigh up the risk-benefit ratio,’” says Alexandra Freeman, executive director of the Winton Centre for Risk & Evidence Communication at the University of Cambridge. But just because most folks aren’t doing math doesn’t mean they’re not chewing on the problem. As Freeman says, “a risk is very subjective.”

So, OK, let’s talk about those blood clots. Freeman’s group put together a bunch of infographics that weaved a few of these threads into a useful tapestry. Instead of comparing the risk of getting Covid to the risk of getting vaccinated—an apples-to-oranges problem—they instead published a document comparing the potential blood-clot risk of the AstraZeneca vaccine to its actual benefit, the number of Covid-related intensive care unit admissions prevented by its use. And then they diced that up by age group and exposure risk. (In real life, exposure risk would differ from country to country and even across professions … and the group assumed 80% efficacy for the vaccine across the board, a necessary simplification … and they used a fixed time span of 16 weeks, because all of these risks shift over time as infection rates wax and wane. Statistics!)

In 100,000 people with low exposure risk, they calculated, the AstraZeneca vaccine might be expected to cause 1.1 people to get blood clots and prevent just 0.8 ICU admissions. If you’re an only-looking-out-for-number-one sort of person, that looks like a reason to avoid the AstraZeneca vaccine—and indeed, European regulators have limited its use. Lucky there’s all those other vaccines.

At the other extreme, among people who for some reason have a high exposure risk—lots of infection running rampant in their county, say—in the 60-to-69 age bracket, the vaccine might cause just 0.2 cases of blood clots (which seem mostly to affect younger people) but keep 127.7 people out of the ICU. It makes a stark case. In most of the Winton Centre groupings, the risk of the AstraZeneca vaccine pays off.

Again, though, the US and Europe ceded the power to evaluate these vaccines to the companies that made them. Each one used slightly different protocols and different populations. A multi-arm study of all of them might have ironed out these statistical kinks. The WHO actually announced such a trial in 2020; nothing seems to have come of it.

For one thing, a multi-arm trial would have made it easier to figure which vaccines’ risks and benefits best match to specific subgroups. That would’ve taken the my-risk/my-benefit hesitancy argument at least partially off the table. “It is wrong to compare vaccines based on relative risk reduction when the studies were done differently,” Olliaro says. “They were done with different protocols, different definitions of what a case is, and completely different populations with different risks.” Put all those vaccines together with the same endpoints and well-understood populations, though, and you get answers to which one is better for whom.

The researchers running my made-up trial might’ve also built in an early, robust look at how each vaccine mitigated milder cases of Covid-19 and its transmission. That would’ve cut deep into the hesitancy arguments.

People might think that their risk of getting Covid-19 isn’t high enough to bother with vaccination—and that in fact the risk of vaccine side effects outweighs it. That argument presupposes a lot. It assumes that an asymptomatic or mild case of Covid-19 is no big deal, and it ignores the way the virus that causes Covid-19 moves from person to person.

First of all, long Covid, the months-long persistence of symptoms, can result from even a mild infection. Nobody understands it, but the possibility of it should factor into those more qualitative, intuitive risk-of-infection calculations that Freeman talks about. Some preliminary work even suggests that women under 60—the same group apparently most at risk of those ultra-rare blood clots linked to the J&J vaccine—are also most at risk of long Covid. But nobody knows how big these benefits might be, to be clear. “I have not seen or even attempted to calculate the benefits of vaccines preventing long Covid,” Olliaro says.

The transmission issue also means reminding people worried about vaccine side effects that if they did get sick, even if it wasn’t bad for them, they could still transmit the disease to loved ones—the opposite of herd immunity, if you will. “If you are infected, you can infect other people,” Olliaro says. “So what you’re not factoring in is the collective benefits of the vaccine.”

None of the trial protocols clearly and directly addressed whether or not the vaccines prevented transmission of the virus; they were focused on stopping severe illness and death to alleviate pressure on the health care system. Now, post-authorization data has begun to show that, yes, the vaccines seem to prevent transmission—and yes, they also prevent milder cases, though not as effectively as severe ones. But the message isn’t out there. If the response to the pandemic has had one lesson, it’s that people with more information make better decisions. You want to get your normal life back? You have to stop the pandemic for other people. The sin of vaccine statistics is making that simple truth murky instead of clear.