Welcome to Limits of Inference. I am happy my blog has become a substack newsletter. This isn’t my full-time gig, so I cannot promise to write on a schedule. Hopefully, that’s ok. I don’t want to crowd your inbox anyway.

Limits of Inference is a newsletter about data generally—and specifically, the way data fails to provide objective truth about the world. I didn’t intend to write about biology when I started this project. However, COVID-19 is dominating my thinking. This topic is a natural one for me since I’ve spent half of my career working in biomedical engineering developing drugs and therapies. My deep dive into problems in biological research spurred my initial interest in the misuse of data. So, the next few articles will be COVID-19 related. Drop me a note in the comments if you want to hear about something else. Also, happy to answer questions.

Subscribe now

Childhood vaccinations allow kids to play together without spreading disease. IUDs allow couples to plan their family and still have a fulfilling sex life. Chemotherapy allows cancer patients more time to see their family and friends. NSAIDs allow athletes to manage inflammation to train harder. Even needle exchange programs allow safer drug use by reducing the risk of infection.

Public health and medicine advanced dramatically in the last century. These success stories share a common trait. Public health policies, particularly progressive ones, find a way to say yes to life.

Yet, in 2020, many public health experts have advocated for policies that deny life. We locked down instead of spending time with friends. Our grandparents died alone instead of being surrounded by their loved ones. Passports were made useless instead of providing opportunities to explore the world. Young adults stopped nurturing new relationships that might someday become their chosen family.

Many people believe these sacrifices are necessary to combat this pandemic. Trusted officials have told the public that science says this is true. But in fact, scientific research aims to design new solutions that allow life to continue despite the ever-present risk from disease. It is not true that we have no other options. Even if it were, it is the job of scientists to invent better options if none exist. A lack of alternatives would reflect a failure of scientists, not a consequence of science.

There are different ways for a leader to respond to a risk. An expert can publish weekly death statistics as haunting reminders that even going outside increases skin cancer risk, or an expert could hand out bottles of sunscreen before a 5k and say, enjoy your run.

An educator can teach abstinence and use the negative consequences of pregnancy and STDs to scare teenagers. Alternatively, an educator can teach about the risks while also demonstrating how to use a condom and engage in safe sex.

Scientists had a choice to lead with progressive policy and even be creative with their solutions during this pandemic. Instead, they chose to lead with fear and motivate with shame. I am proud of the history of this field, but this year has been a disaster. It is time to stop the fear-mongering and start being better scientists and better doctors. It is time to find a way to start saying yes to life.

Policy failures often stem from inadequate definitions of success. I think this is a root cause of the pandemic failures in 2020. As I have written before, we’ve focused on the wrong data, wasting resources trying to prevent infections instead of minimizing disease. There is more.

To start, let’s clear this one up. Public health and medical professionals are not in the business of saving lives. The reason is simple: every single one of us is mortal. Every person who has been born will die. If one tries to measure any policy’s success using the metric of “lives saved,” there is only one possible conclusion— we have saved zero lives, ever.

To use ‘lives saved’ as a success metric, a scientist or analyst must cherry-pick the data, limiting the scope of a graph or metric to one cause of death or a period of time. This analytical decision can be deceptive if used to hide the counterfactual. A policy that prevented deaths by one cause necessarily increased the number of deaths by another. A discussion of what happened instead is the context necessary to judge if a policy is effective overall.

The claim is this: lockdowns, social distancing, and travel bans saved lives in 2020 by preventing the spread of COVID-19. This claim is dubious, given that the virus is spreading virtually uncontrolled despite global lockdowns. Let’s assume these policies at least helped toward this goal.

This is what happened instead. Lockdowns, travel bans, social distancing, and the fear-based messaging required to get public acceptance of these interventions disrupted supply chains. The social isolation from shutdowns increased poverty, depression, and obesity, all risk factors for life-long disease. Told to stay home, most people avoided the doctor’s office or hospital altogether.

This is what these second-order consequences look like for us today. The world anxiously awaits vaccines, but Pfizer cannot manufacture their vaccine at a sufficient scale. They do not have enough equipment, such as cold storage. Why? We told the people who make the freezers back in March—stay home to save lives. With long lead times required to manufacture complex equipment, we couldn’t ramp up fast enough to get the COVID-19 vaccine to the population that needs it.

Some nurses and doctors on the front line are at greater risk from infectious disease because they do not have adequate PPE (masks, gloves, gowns). We do not have enough PPE because, again, we told the people who make the PPE —stay home to save lives.

Supply problems were the inevitable consequence of telling people to stop working. Global supply chains are enormous. Millions of people need to work in some capacity or another to manufacture even a single N95 mask end-to-end. Medicines and supplies that doctors use to treat diseases exist because people leave their houses to work together. A functional economy is a boon to public health. Shutting it down has real negative consequences even to health and medicine.

Then, there is the fact that social distancing itself is not good for anyone’s health. In its extreme, solitary confinement is a punishment, even torture. Because it is mentally straining, social distancing encourages terrible habits such as sedentary lifestyles, binge eating, drug use, and excessive drinking. If someone never loses weight, never changes their lifestyle, or never kicks the depression caused by this quarantine, it could mean a lifetime of poor health, including early death (but it won’t be observed in 2020).

Even if we consider diseases that occurred just this year, evidence of the pandemic response policy's negative consequences is ubiquitous. There are hundreds of thousands of missing cancer diagnoses in the US. When diagnosed, the cancer is late-staged and less likely to be treatable. Diseases of poverty rose as travel bans closed hospitals staffed by foreign doctors. There were an estimated 200,000 missing tuberculosis diagnoses due to just the first three months of travel bans this year. Patients in the US are 2.4 times more likely to die from a heart attack in 2020 than in 2019. Organ donations have dwindled this year, endangering the lives of thousands of people waiting for a transplant.

Even deaths now counted as caused by COVID-19 may be caused indirectly by social distancing. Around ten thousand additional Alzheimer's patients have died this year, making dementia patients one of the most at-risk groups for the pandemic. Social distancing lowers the chance that a person gets exposed to SARS-COV-2 (benefit). However, social isolation also speeds up Alzheimer’s and dementia progression (cost). Alzheimer’s and dementia have a one hundred percent fatality rate. When dementia progresses, it increases the risk of death from viral disease. In short, social distancing might decrease the odds of getting COVID-19, but it increases the odds that a dementia patient will die from COVID-19 (and every other disease). It is not possible to disambiguate these effects from just the data on a death certificate. As fewer people died from the flu this year, the numbers appear close enough that we may find someday that more of the excess dementia deaths resulted from social distancing than from the novelty of the virus.

None of these consequences are being considered in the design or execution of pandemic policy today. Cities are opening and closing based on the number of COVID-19 infections alone. Monitoring the spread of COVID-19 is essential, but it is one of the many health risks we face. This narrow focus on one disease is resulting in troublesome and self-serving storytelling with data from professionals.

Today, the CDC (and consequently many journalists) refer to these deaths as “COVID-related” despite people not dying of that disease. Some analysts count these numbers as deaths from the pandemic itself, usually applying an analytical approach that attributes all excess deaths to the pandemic as a default assumption as if human failures could not possibly contribute.

A reminder of the limits of inference — data tells us that something happened (it tells us someone died), not why (was it the pandemic or the policy?). The why is a story we tell ourselves to make good decisions in the future. The ‘why’ is inference. A better story results in better policy.

It is convenient for the CDC to use the phrase “COVID-19 related” because it sounds to the casual reader that these deaths were the inevitable and unfortunate consequence of the pandemic. An alternative story is that fear-mongering, a single-minded focus on one disease, caused elderly patients to be too afraid to go to the hospital for heart attack symptoms or get preventative care. Telling older people to be frightened or families to stay apart was a policy choice. Hospitals were never full. Therefore, I believe a more accurate data story counts these deaths as costs caused by the CDC and NIH —lockdowns saved some people and killed others. By acknowledging this more nuanced data story, we could have made better decisions.

Some deaths this year were inevitable. We are living through a deadly pandemic, and we are all mortal. I believe many of the second-order consequences of pandemic policy were avoidable if we thought more holistically about defining success and managing risk. Other policy options allowed life to continue in the spirit of sunscreen and safe-sex.

We could have told the public to limit social activities to the one or two they value most (and keep it outdoors when possible). This policy would allow people to live well while still flattening the curve and keeping hospital beds open.

We could have told workers (accurately) it was safe to keep working unless they are in a high-risk group. This choice would have kept supply chains and economies going while still protecting the most vulnerable. We could have even done more work, manufactured supplies more quickly, to get the vaccines distributed faster. We could have created operations centers and medical transportation teams to make sure individual hospitals were never overwhelmed.

With a message of relative risk, it would have been easy for the public to understand why experts actively encouraged even high-risk groups to get preventive care like annual check-ups and cancer screenings from their doctors. COVID-19 is dangerous, but so is cancer. It is always important to live each day that we have on earth and take care of ourselves.

We could have asked everyone to wear a mask as a low-cost intervention. Like condoms, masks are not fun but not a huge detractor from life. I doubt people would have rebelled as much it didn’t feel the seeing people’s faces was the last hint of normal social interaction.

We could have compensated nursing home workers well and scheduled shifts, so they quarantined monthly. This way, the disease did not enter the facilities where 40% of all deaths have occurred. Instead, we told everyone, including nursing home workers, to quarantine indiscriminately and without any compensation.

These are just a few different policy ideas that reduce risk while allowing more life. Public health should work by looking at the totality of harm to life, not just the graph of one disease over one year. COVID-19 is a risk, but it is not the only one we face.

One would hope that the CDC did the analysis and decided that more people needed to die of heart attacks, cancer, tuberculosis, drug addiction, strokes, kidney failure, and Alzheimer’s disease (…the list continues) to prevent the spread of COVID-19. I don’t think they did. I think these scientists never thought about second-order consequences. They considered the one disease given the data in the one graph. This hypothesis comes from my experience working in the field and watching the professionals leading the response today. That’s the topic for a future post.

The point here is that the story isn't as simple as ”stay home to save lives.” Zero lives have been saved this year (or any year). However, billions of years of life have not been lived. Pandemics are real. Risk is never zero. However, the science supporting lockdowns as a response to a pandemic is only obvious to those who cherry-pick the data and deprioritize most of the population's health needs to tell the story.

We should start counting everyone, everyone’s life, and everyone’s health when we define success. It would tell a better story, be more consistent with data, and we would make better decisions.

Share

Limits of Inference

The best and worst of public health