Controversial Paper Claims COVID-19 "Lockdowns" Had Little Public Health Effect

What impact did lockdowns and other restrictions have on public health during the COVID-19 pandemic? According to a preprint paper from researchers with the Johns Hopkins Institute for Applied Economics, Global Health, and the Study of Business Enterprise, little to no public health effects.

However, the meta-analysis of 24 studies has not undergone peer review, and the findings are drawing criticisms. The researchers grouped the studies into 3 groups: lockdown stringency index studies, shelter-in-place order (SIPO) studies, and specific nonpharmaceutical intervention (NPI) studies.

Part of the criticism over the study is related to the definitions used. Neil Ferguson, OBE, FMedSci, an epidemiologist and director of the MRC Centre for Global Infectious Disease Analysis, Jameel Institute, Imperial College London, noted that “the policies which comprised ‘lockdown’ varied dramatically between countries, meaning defining the term is problematic.”

The studies included in the meta-analysis attempted to establish a relationship between mortality and lockdown policies. The authors noted they were evaluating mandatory NPIs and did not look at the effect of voluntary behavioral changes, recommendations, or governmental services. Among the studies excluded were:

• Studies that used cases, hospitalizations, or other measures
• Interrupted time series studies comparing mortality rates before and after lockdowns
• Studies analyzing the effect of early lockdowns vs later lockdowns
• Synthetic control method papers

A total of 34 papers were eligible for review—ultimately, 24 were included in the meta-analysis. Of the 34 total papers, 12 were also working papers and had not undergone peer review. The most recent data in all 34 studies were from April 2021. The Delta variant began widely circulating in the United States and Europe after this time.

There were 7 studies examining lockdown stringency, and the authors that, on average, the lockdowns reduced mortality from COVID-19 by just 0.2%. Just 1 of the 7 studies found a significant and substantial effect. The remaining 6 studies found little to no effect.

There were 13 studies estimating the effect of SIPOs. Of these, 7 looked at multiple NPIs, with SIPO being just one, and 6 studies analyzed SIPO vs no SIPO in the United States. Based on the results of these studies, the authors concluded that SIPOs reduced COVID-19 mortality by 2.9%.

Finally, 11 studies looked at multiple specific NPIs. The definition of specific NPIs varied from study to study—for instance, one study reviewed nonessential business closures and bar/restaurant closures, while another study reviewed business closures but no bar/restaurant closures—making a comparison difficult, the authors noted.

Based on the studies reviewed, the authors concluded “generally there is no evidence of a noticeable relationship between the most-used NPIs and COVID-19.”

Mandating face masks had a large effect of reducing COVID-19 mortality by 21.2%, according to 2 studies. School closures reduced mortality by 4.4%, and lockdowns and limiting gatherings actually seemed to increase mortality modestly (by 0.6% and 1.6%, respectively). Finally, the findings on business closures were mixed with 3 studies finding little to no effect and 3 studies finding large effects.

Another expert from Imperial College London, Samir Bhatt, also cautioned careful interpretation of the findings. Bhatt, a professor of statistics and public health, also pointed to the definition issue. “The authors define lockdown as ‘as the imposition of at least one compulsory, non-pharmaceutical intervention,’” he said. “This would make a mask wearing policy a lockdown. For a meta-analysis using a definition that is at odds with the dictionary definition (a state of isolation or restricted access instituted as a security measure) is strange.”

He added that the method to estimate the impact of NPIs was not accurate. “Such interventions are intended to reduce contact rates between individuals in a population, so their primary impact, if effective, is on transmission rates,” he said. The impact of NPIs on hospitalization and mortality can be delayed for up to several weeks and mortality immediately following the introduction of lockdowns.

“Neither is lockdown a single event as some of the studies feeding into this meta-analysis assume; the duration of the intervention needs to be accounted for when assessing its impact,” said Ferguson. Notably, Ferguson coauthored a paper in 2020 that utilized a model simulation to predict that a lockdown would reduce COVID-19 mortality substantially, as well as another paper—along with Bhatt—evaluating the effect of major interventions across 11 European countries, which estimated only 4% or less of the population across all 11 countries were infected with the virus and that NPIs had “a large effect on reducing transmission.”

The authors of the meta-analysis did note that there may be objections to their findings because they do not look at the role of timing. “If timing is very important relative to strictness, this suggests that well-timed, but very mild, lockdowns should work as well as, or better than, less well-timed but strict lockdowns,” they wrote. However, they added it is difficult to analyze the data supporting the timing thesis.

The authors of the Johns Hopkins study are economists, not epidemiologists, and their ultimate conclusion is related to that expertise. They wrote that the lockdowns during the first wave of the pandemic “have had devastating effects.”

“They have contributed to reducing economic activity, raising unemployment, reducing schooling, causing political unrest, contributing to domestic violence, and undermining liberal democracy,” the authors concluded. “These costs to society must be compared to the benefits of lockdowns, which our meta-analysis has shown are marginal at best. Such a standard benefit-cost calculation leads to a strong conclusion: lockdowns should be rejected out of hand as a pandemic policy instrument.”

All 3 authors are also from Sweden and Denmark. Sweden famously decided not to implement any lockdowns and never mandated, only recommended, masks in public. The country recorded more cases per capita and deaths than neighboring countries: Reported deaths were 3 times more than Denmark, 8 times more than Finland, and 10 times more than Norway. A December 2021 article in BMJ said public health experts and scientists believed “Sweden’s situation remains precarious.”

Both Sweden and Denmark just this week announced they are lifting all COVID-19 restrictions as the pandemic is no longer burdening hospitals. However, Denmark’s decision occurred despite the number of cases increasing.