Eli Cahan is a medical student at NYU on leave to complete a master’s in health policy at Stanford as a Knight-Hennessey Scholar. His research addresses the effectiveness, economics, and ethics of (digital) health innovation.
More posts by this contributor
Day after day, the burden of COVID-19 caused by SARS-CoV-2 mounts further. As of this writing, nearly 400,000 patients worldwide were confirmed for the disease, including over 46,000 cases spanning every state in this country.
In tandem with this mounting burden, due to numerous fumbles over the past 10 weeks, the U.S. has faced significant bottlenecks in the production of diagnostic testing and imposed substantive red tape to deter testing. As more diagnostic tests have come online, the rate of “confirmed cases” — the key performance indicator (KPI) being monitored by decision-makers across the country — continues to accelerate.
Why businesses love KPIs
Leaders of startups are familiar with the attention paid towards KPIs such as profit margins, burn rates, net dollar retention rate, and customer acquisition costs. These metrics, when chosen appropriately, allow leaders to continuously take the pulse of their companies and take action in response.
Along these lines, Phil Nadel has written on TechCrunch that founders “cannot hope to grow a company in any meaningful way without…KPIs…[b]ecause KPIs, if constructed correctly, give management and potential investors a cold, analytical snapshot of the state of the company, untainted by emotion or rhetoric” (emphasis added).
Conversely, when misconstructed, misconstrued or overlooked, KPIs can cause organizations to crash and burn. This phenomenon has been deemed “surrogation,” or reflecting the potential for critical thinking around strategy to be subverted by elevation or suppression of a single number. For example, maximizing profit margin can hurt the quality of goods/services, and maximizing lifetime value (LTV) can negatively impact customer experience by, say, encouraging the use of shady cross-selling or up-selling techniques that are harmful in the long-term to customers (as was the case at Wells Fargo after the financial crisis).
Taken too far, wrongly-selected KPIs can cause organizations to suffer profoundly, such as with Uber’s patchy quality standards for driver recruitment leading to innumerable controversies around customer experience. And while consistently poor quarterly failures in the boardroom can lead to organization crises, they do not bear remotely the same stakes as public health crises.
The primary KPI for COVID-19 has been an unreliable figure in the U.S.
During the COVID-19 pandemic, the primary KPI tracked by U.S. leaders has been the number of confirmed cases of the disease. Yet epidemiologists monitoring the outbreaks have become increasingly frustrated with the meandering pace of actions taken by U.S. decision-makers. The core of the mismatch between actions desired by scientists and those (not) taken by policymakers lies in misinterpreting the “denominator.” In other words, misconstruing the pandemic’s primary KPI.
The denominator refers to the formula from which mortality rates are calculated — the number of deaths divided by the number of cases. While this seems to be straightforward algebra, the devil is in the details. Since the beginning of the outbreak, the Chinese government has waffled on their definition of “confirmed cases”: deciding ultimately to go with a definition (positive only if laboratory-confirmed, regardless of symptoms or other tests) that may suppress the real number. Analogously, diagnostic testing snafus in the U.S. suggest that the number of “confirmed cases” here is simply not reliable. Both of these stand in stark contrast to countries such as Taiwan and South Korea, both of which swiftly diverted resources to scale up, broaden, and fully report testing.
Predictably, these snafus have led to diagnosis of disproportionately severe cases thus far in the U.S.: with the exception of NBA players and movie stars, only highly symptomatic individuals have been eligible to receive testing (due to clinical red tape), able to receive testing (due to shortages), and subsequently confirmed. This selection bias of severe cases implies that overall case number — the true denominator — is markedly underestimated. Accordingly, since the beginning of February, epidemiologists have been vocal that “simple counts of the number of confirmed cases can be misleading indicators of the epidemic’s trajectory.”
Relying on confirmed cases as the primary KPI may have delayed and misled critical action
A timeline of COVID-19’s progression in the U.S. illustrates the impact of using confirmed cases as the country’s primary KPI for public health response.
On January 22, President Trump stated that “we have [SARS-CoV-2] totally under control, it’s one person [confirmed].” On February 25, with 53 confirmed, he claimed that “the coronavirus…is very well under control in our country. We have very few people with it.” The following day, the president declared that “the risk to the American people remains very low.” On March 6, with 227 confirmed, he shared that “I think we’ve done a tremendous job of keeping [the number of confirmed cases] down.” And just one week ago on March 17, the president praised West Virginia as the single state without any cases, hailing “Big Jim, the governor…must be doing a good job.”
By the end of that day, the praise no longer held. Reports later emerged that the reason West Virginia had no cases for so long was related to the fact that the state was ill-equipped, and perhaps resistant, to performing testing that might come back positive.
With confirmed cases on U.S. soil as the North Star for decision-making — despite “the system blinking red” on other dimensions — early decisions to take vital preventive actions were punted. It wasn’t until March 13 (over seven weeks after the first confirmed U.S. case) that the president declared a national emergency. And even as 70 million Americans were under lockdown as of March 20, 47 states maintained few restrictions for social distancing (which remains the only intervention currently with proven effectiveness against COVID-19). Those under lockdown permit leaving for “essential” tasks, which itself is fuzzily-defined.
Moreover, emerging evidence suggests that mild (“subclinical”) cases — exactly the ones that have not been tested — may be driving community spread of the virus. Nonetheless, focus on confirmed cases as principal KPI has elicited few interventions against these “below the surface” cases. Domestic travel restrictions by foot, bike, car, bus, train, and plane remain mostly limited. Spring breakers continue to party at Miami Beach’s hookah shops and Nashville’s honky-tonks.
Simultaneously, numerous universities — sensitive to the fact that college dormitories are amongst the highest risk settings for contracting an infection — suspended classes and evacuated their students. However, these moves overlook the fact that college students are amongst the least likely to manifest symptoms. As such, by seeking to prevent any confirmed cases on their campuses, these universities may have ushered the virus into the homes of parents and grandparents in innumerable local communities across the U.S.
Alternative KPIs for COVID-19 can help get the U.S. back on track
If the wrong KPI created this mess, then the right KPI(s) are urgently needed to begin fixing it.
For starters, risk stratification is desperately required. Rather than resorting to black-and-white measures of infected versus uninfected, risk spectra can better characterize the threat faced by specific individuals, communities, states, and countries.
For example, mortality risk can be quantified and monitored. Elderly individuals and those with pre-existing medical conditions are at the highest risk of severe infection. By deriving and pooling individual risks, the relative threat to communities can be evaluated: encouraging high-risk communities to take more immediate and more proactive preventive action. Florida, for instance, could benefit from this kind of measurement: 27% of the state’s residents population are elderly, but thus far shockingly few individuals have been tested for coronavirus in retirement communities like the Florida Keys (where 75 people had been assessed as of March 24 amidst the height of Spring Break). This could lead to protective policies like visitation restrictions and hygiene guidelines.
Additionally, transmission risk can be a useful KPI. Certain demographic groups, like students and healthcare workers, are at much higher risk of propagating the virus — with or without the presence of symptoms. Individuals with exposure to the virus in confined spaces (such as cruise ships or airplanes) also have much higher transmission risk. Scoring metrics could be adapted from other infectious diseases to help policymakers better visualize and prevent transmission in their communities. This could encourage prospective procedures like contact tracing and symptom monitoring.
Designating mortality risk (for vulnerable patients) and transmission risk (to vulnerable patients) as KPIs could provide much more granularity to decision-makers than can confirmed cases alone. Including these collectively on a dashboard will generate insights and stimulate further actions than relying on a single narrow, fallible KPI.
And of course, these are only two (reductive) examples of potentially useful metrics for public health responders. Far more diverse forms of metrics are possible. These are often pioneered, unsurprisingly, by startups — such as in the case of Kinsa Health, a producer of smart thermometers. The company’s temperature geo-maps are already providing a leading COVID-19 indicator for local decisionmakers, as they have previously for the seasonal flu (compared to CDC models).
In healthcare and beyond, new KPIs will prove critical moving forward
As far as COVID-19 goes, the steps our country needs to take are fairly clear. An analysis by researchers at Imperial College London’s COVID-19 Response Team suggested that near-universal social distancing measures are the only tool available to prevent “hundreds of thousands of deaths and health systems (most notably intensive care units) being overwhelmed many times over.” Countries that continue to be ravaged by the virus (such as Iran and Italy) enacted social distancing interventions excessively late. As the trajectory of the infection curve in the U.S. closely mirrors these countries (despite probable underestimates of case burden), more universal measures are sorely needed. Better healthcare KPIs can help us understand the next steps we need to take.
As far as the startup community goes, there is much to learn from this situation. A proverb in medicine advises: “during a cardiac arrest, the first procedure is to take your own pulse.” Since startups may face numerous crises (in the business sense) amidst the economic convulsions of COVID-19, composure will be essential to guide decisions under pressure. Meaningful KPIs are the stethoscopes, tourniquets, and barometers that enable startups to take their own pulse—and to rapidly commence any resuscitation that may be required.
Another medical proverb dictates that “an ounce of prevention is worth a pound of cure.” Prescient leaders of startups would be well-served by seeking the tools for prevention sooner rather than later.
