2022-zelenko-paper[92250].pdf

The June 2020 Zelenko case series [17] is reported in a letter that Zelenko sent to the Israeli Health Minister at the time, Dr. Moshe Bar Siman-Tov, on June 14, 2020, which was later made publicly available. In the letter, Zelenko reported that a total of approximately 2,200 patients were seen as of June 14, 2020, with 800 patients deemed high-risk under the same criteria and treated with the triple-drug therapy, since the beginning of the pandemic. The reported cumulative outcomes are, 12 hospitalizations, 2 deaths, no serious side effects, and no cardiac arrhythmias. During the April 2020–June 2020 interval, Zelenko enhanced his triple drug therapy protocol with oral dexamethasone and budesonide nebulizer at the beginning of May 2020. He introduced the blood thinner Eliquis towards the end of May 2020 and beginning of June 2020. Ivermectin was not used by Zelenko until October 2020. Consequently, the DSZ study [2] and the Zelenko April 2020 case series [16] reflect the outcomes of the triple drug therapy, when used by itself as an early outpatient treatment. The Zelenko June 2020 case series [17] includes the use of steroid medications and a blood thinner, so the underlying treatment protocol is closer to the McCullough protocol [21–23]. It is worth noting that both letters [16, 17] were originally posted on Google Drive by Zelenko and were censored by Google during 2021.

The April 2020 letter [16] was cited by Risch [64], whose paper has also preserved the corresponding case series data. The June 2020 letter case series data [17] was independently reported by a subsequent publication by Risch [81], however, it included only the number of reported deaths, and not the number of hospitalizations. The authors have attached copies of all three Zelenko letters [16, 17, 24] to our supplementary material document [25]. The Procter case series were reported consecutively in two publications [18, 19]. The first paper [18] reports on 922 patients that were seen between April 2020 and September 2020, of which 320 were risk stratified as high-risk patients and treated with the McCullough protocol [21–23]. The outcome was, 6 hospitalizations and 1 death. The second paper [19] reports on an additional patient cohort seen between September 2020 and December 2020. Out of the total number of patients during that time period, 549 were risk stratified as high-risk and treated with an outcome of 14 hospitalizations and one death. For both case series, the risk stratification criteria were similar to those used by Zelenko. However, the age threshold used to risk stratify patients as high-risk was lowered to 50 years.

The medications used were customized for each patient in accordance with the McCullough protocol [21–23] and included hydroxychloroquine, ivermectin, zinc, azithromycin, doxycycline, budesonide, foliate, thiamin, IV fluids, and for more severe cases, dexamethasone and ceftriaxone were also added. Demographic details for the cohorts were reported in the respective publications [18, 19]. The final high-risk patient case series is extracted from a recent cohort study [20] of 10,429 patients that were seen between March 2020 and December 2020 by Raoult’s IHU Méditerranée Infection hospital in Marseille, France. From the entire cohort, 8,315 patients were treated with hydroxychloroquine, azithromycin and zinc. Of those patients, those

12 Case series

Total High-risk Hospitalizations & Deaths

DSZ study [2] 712 141 Zelenko 04/2020 [16] 1450 405 Zelenko 06/2020 [17] 2200 800 Procter I [18] 922 320 Procter II [19] ? 869 Raoult [20] 10429 1495

4 (2.8%) 1 (0.7%) 6 (1.4%) 2 (0.4%) 12 (1.5%) 2 (0.25%) 6 (1.8%) 1 (0.3%) 20 (2.3%) 2 (0.2%) 106 (7.0%) 5 (0.3%)

Control groups

Total High-risk Hospitalizations & Deaths

DSZ control [2] Israeli control [60] Raoult control [20]

377 < 377 4179 < 4179 2114 520

58 (>15%) 13 (>3.4%) N/A 143 (>3.4%) 38 (7.3%) 11 (2%)

FIG. 4: Case series list: The table lists the total number of patients, the subset of high risk patients that were treated with a sequenced multidrug regimen, number of patients that were hospitalized, and number of deaths, for the following case series: Derwand-ScholtzZelenko study treatment group [2], Zelenko’s complete April 2020 data set [16], Zelenko’s complete June 2020 data set [17], Procter’s observational studies [18, 19], and Raoult’s high risk (older than 60) treatment group [20]. The table also lists the same data for the control group in the DSZ study [2], the untreated group in the Israeli study [60], and the control group in the Raoult study [20].

older than 70 or with comorbidities were also treated with enoxaparin and low-dose dexamethasone was given on a case by case basis to patients that presented with inflammatory pneumonopathy, high viral loads, or on a case by case basis. This treatment protocol is consistent to some extent with the principles that underlie the McCullough protocol [21–23]. The remaining 2,114 patients did not receive hydroxychloroquine or azithromycin or both because it was either contraindicated or because the patients did not consent to using one or two of these medications. This cohort was used in the Raoult study [20] as a control group. The study risk-stratified the patients by age (see Table 1 of Ref. [20]) making it possible to extract a case series of high-risk patients under the restriction age ≥ 60. In the treatment group, this results in 1,495 highrisk patients with 5 deaths and 106 hospitalizations. In the control group, under the age ≥ 60 constraint, there are 520 high-risk patients with 38 hospitalizations and 11 deaths. The authors note that no serious adverse events to the medications were reported and that the reported deaths were not related to side effects of hydroxychloroquine or azithromycin. Furthermore, no deaths were reported for age < 60 cohort in both the treatment group and control group. 4.2.

Tabular summaries of the Zelenko, Procter and Raoult case series

Fig. 4 summarizes the aforementioned case series, including the treatment groups from the DSZ study [2], the Zelenko [2, 16, 17] and Procter [18, 19] case series and the age ≥ 60 treatment group from the Raoult study [20]. Note that the Zelenko June 2020 case series and the Procter II case series as reported on Fig. 4, combine the two respective consecutive

Study

odds ratio 95% CI

p-value

Exact Fisher tests on mortality rates DSZ study vs DSZ control 0.2 Zelenko 04/2020 vs DSZ control 0.13 Zelenko 06/2020 vs DSZ control 0.07 DSZ vs Israeli control 0.2 Zelenko 04/2020 vs Israeli control 0.14 Zelenko 06/2020 vs Israeli control 0.07

0.02–1.54 0.12 0.03–0.61 0.003 0.01–0.31 10−5 0.03–1.45 0.09 0.03–0.57 0.0002 0.02–0.28 10−9

Exact Fisher tests on hospitalization rates DSZ vs DSZ control Zelenko 04/2020 vs DSZ control Zelenko 06/2020 vs DSZ control

0.16 0.08 0.08

0.05–0.45 0.02 0.03–0.19 10−13 0.04–0.16 10−19

FIG. 5: Exact Fisher test comparing the mortality rate reduction and hospitalization rate reduction between the high risk patient treated group the DSZ study [2], Zelenko’s complete April 2020 data set [16], and Zelenko’s complete June 2020 data set [17] against the low risk and high risk patient control groups in the DSZ study [2] and the Israeli study [60]. The p-values where there is a failure to establish 95% confidence are highlighted.

case series. We also report on Fig. 4 the DSZ study’s control group [2], the alternative Israeli control group [60] and the age ≥ 60 part of the Raoult control group [20]. We emphasize that all reported treatment group case series consist of high-risk patients. From a cursory examination of Fig. 4, we see that the mortality rate is consistent across all treatment groups, which speaks to the consistency Bradford Hill criterion [76]. Hospitalization rates are also consistent between the Zelenko [2, 16, 17] and Procter case series [18, 19], but there is a clear discrepancy with the hospitalization rates reported in the Raoult treatment case series [20]. We believe that the reason for the discrepancy is that both Zelenko and Procter explicitly aimed to prevent hospitalizations due to the poor outcomes of the inpatient treatment protocols used in the United States. In Marseille, France, Raoult had the option of using his IHU Méditerranée Infection hospital, for short hospitalizations, in order to closely monitor his more concerning cases. In Fig. 5, we show the results of comparing the Zelenko April 2020 [16] and Zelenko June 2020 [17] case series against both the original DSZ control group [2] as well as the alternative control group from Israel [60].

Although in the original DSZ study [2] mortality rate reduction was not statistically significant, we have found that comparing either the Zelenko April 2020 case series [16] or the June 2020 case series [17] against either control group, gives more than 90% mortality rate reduction, which is also statistically significant in terms of both p-value and confidence interval. Likewise, we see at least 90% hospitalization rate reduction when the Zelenko April 2020 case series or Zelenko June 2020 case series is compared against the DSZ control group, which is statistically significant as well. Because the control groups consist of a combination of both low-risk and high-risk patients, whereas the treatment groups consist of only high-risk patients, the re-

13 Study

95% threshold 99% threshold 99.9% threshold

Mortality rate efficacy thresholds DSZ study 3.8% (9.2%) 5.3% (12.8%) 7.0% (14.6%) Zelenko 04/2020 1.8% (4.0%) 2.4% (5.2%) 2.9% (6.9%) Zelenko 06/2020 1.0% (2.0%) 1.2% (2.7%) 1.6% (3.7%) Procter I 1.7% (4.1%) 2.3% (5.8%) 3.1% (7.8%) Procter II 0.84% (1.82%) 1.08% (2.46%) 1.4% (3.37%) Raoult 0.79% (1.40%) 0.96% (1.87%) 1.18% (2.46%) Hospitalization rate efficacy thresholds DSZ study 7.0% (12.7%) Zelenko 04/2020 3.2% (5.4%) Zelenko 06/2020 2.7% (4.2%) Procter I 4.1% (7.3%) Procter II 3.6% (5.2%)

8.8% (17.5%) 3.9% (7.2%) 3.0% (5.0%) 4.9% (9.1%) 4.0% (6.1%)

10.6% (21.5%) 4.7% (9.5%) 3.5% (6.4%) 5.9% (11.6%) 4.5% (7.5%)

FIG. 6: Mortality and hospitalization rate reduction efficacy thresholds, defined as the upper end of the Sterne interval [71], corresponding to 95%, 99%, and 99.9% confidence, for the DSZ study treatment group [2], Zelenko’s complete April 2020 data set [16], Zelenko’s complete June 2020 data set [17], Procter’s observational studies [18, 19], and Raoult’s high risk (older than 60) treatment group [20]. In parenthesis, we also display the corresponding higher selection bias thresholds.

sulting comparisons are biased towards the null, and are thus underestimating the actual efficacy of the respective treatment protocols. This comparison is compelling, due to the consistency between the two control groups, as evidence in favor of showing the existence of treatment efficacy. However, methodologicaly, it may not be convincing enough by itself in terms of measuring the extent of treatment efficacy. We have also calculated the efficacy threshold for mortality rate reduction and hospitalization rate reduction corresponding to the case series by Zelenko [2, 16, 17], Procter [18, 19], and Raoult [20]. The calculations are shown in the supplementary material document [25]. The results are tabulated in Fig. 6. We display the efficacy thresholds for 95%, 99% and 99.9% confidence, which are calculated as the upper end points of the corresponding Sterne interval [71] and, in parentheses, we display the corresponding selection bias thresholds. We use precision of 0.1% for most case series, except for the two largest ones, Procter II [19] and Raoult [20], where we use 0.01% precision. Each threshold corresponds to a mathematically rigorous conditional statement about rejecting the null hypothesis that the corresponding early outpatient treatment protocol is ineffective.

For example, the 1.8% efficacy threshold corresponding to 95% confidence for rejecting the null hypothesis in the Zelenko April 2020 case series [16] corresponds to the following statement: if the expected mortality rate for an equivalent cohort without early outpatient treatment exceeds 1.8%, then the null hypothesis can be rejected with at least 95% confidence. Similar statements can be formulated for each efficacy threshold metric on Fig. 6. Likewise, the 4.0% selection bias threshold for the Zelenko April 2020 case series [16] corre-

sponds to the following statement: If the observed mortality rate, at the population level, for high risk patients, classified as such using the same selection criteria as in the treatment case series, exceeds 4.0%, then we can be 95% confident that the observed signal of efficacy cannot be attributed solely to random selection bias, and we can also reject the null hypothesis with at least 95% confidence. Similar statements are implied from all of the other selection bias thresholds, reported on Fig. 6. These statements are mathematical facts. However, to complete the inference argument, they need to be paired with an inevitably subjective statement that provides an estimate, or at least a lower bound, on the expected mortality or hospitalization rates of similar cohorts without early outpatient treatment. Secondarily, we need an inference about the intervals of mortality or hospitalization rates, in the absence of early outpatient treatment, in order to do the Bayesian adjustment of the efficacy thresholds.

4.3.

Analysis of mortality rate reduction efficacy

To establish that early treatment protocols result in mortality rate reduction, when administered to high-risk patients, we recall that patients have been classified as high-risk based on the following three categories: (1) old age; (2) comorbidities or obesity (with BMI ≥ 30kg/m2 ); (3) shortness of breath upon presentation. The age threshold for high risk classification is age ≥ 60 for the Zelenko [2, 16, 17] and Raoult [20] case series, and age ≥ 50 for the Procter [18, 19] case series. The high-risk treatment groups for the Zelenko [2, 16, 17] and Procter [18, 19] case series include the demographic distribution of all three categories of high-risk patients, whereas in the Raoult [20] case series we have included only age ≥ 60 patients. Our approach, in the following, is to lower bound the mortality rate, in the absence of early outpatient treatment, separately for each of the three high-risk patient categories. Then, the common lower bound becomes applicable to any demographic distribution of the three categories. To establish the existence of treatment efficacy, it is sufficient for this lower bound to exceed the corresponding thresholds of Fig. 6. In the following, we shall now consider the mortality rate for each of the three high-risk patient categories separately.

With regards to the first category of patients classified as high-risk due to old age, the earliest data from China [59] as of February 11, 2020, estimated a minimum of 3.6% mortality rate for patients older than 60 and a minimum of 1.3% mortality rate for patients older than 50 (see Fig. 7). These numbers are corroborated with numbers from China [57] and Italy [58] as of March 17, 2020 (see Fig. 8). We can also estimate the mortality risk of the first category of high risk patients (age ≥ 60 or age ≥ 50) using adjusted estimates by the CDC [61–63] of COVID-19 deaths per symptomatic cases. The CDC report attempts to adjust for the differences in underreporting of symptomatic illness, hospitalizations, and deaths, and it is based on reports ranging from February 2020 to September 2021. The raw data and a copy of the CDC Report website are given in our supplementary material doc-

14 Age Deaths Cases

CFR

10-19 20-29 30-39 40-49 50-59 60-69 70-79 ≥ 80

0 416 0% 7 3619 0.193% 18 7600 0.237% 38 8571 0.4% 130 10008 1.3% 309 8583 3.6% 312 3918 7.96% 208 1408 14.8%

≥ 60

829 13909 5.96%

FIG. 7: Crude Case Fatality Rate data, in the absence of early outpatient treatment, based on early data from China as of February 11, 2020, and published on March 30, 2020. [59]

Age Italy CFR China CFR 0-9 10-19 20-29 30-39 40-49 50-59 60-69 70-79 ≥ 80

0% 0% 0% 0.3% 0.4% 1.0% 3.5% 12.8% 20.2%

0% 0.2% 0.2% 0.2% 0.4% 1.3% 3.6% 8.0% 14.8%

FIG. 8: Crude Case Fatality Rate data, in the absence of early outpatient treatment, based on early data from China and Italy as of March 17, 2020 and published on March 23, 2020 [57, 58].

ument [25]. From that, we calculate for the age ≥ 50 group a mortality rate of 2.26% (95% CI: 1.94% – 2.61%). We cannot deduce an age ≥ 60 mortality rate from the CDC data, but note that the age ≥ 65 mortality rate, according to the CDC is 4.79% (95% CI: 4.11% to 5.52%). The second category of high risk patients are patients with comorbidities regardless of age. In Fig. 9, we show case fatality rates with respect to comorbidities (i.e. cardiovascular disease, diabetes, respiratory disease, hypertension, cancer), based on data from China [57] in the period up to February 11, 2020, and additional data from Israel [60] with patients diagnosed in the period up to April 16, 2020, and deaths recorded up to July 16, 2020. There is variability in mortality rates from 5% to 15%. The Israeli data appear to show higher mortality rates than the data from China, and the reason for that could be that the Israeli study [60] accounted for the time lag between patient diagnosis and death. These studies do not account for the mortality risk from obesity and also do not account for the mortality risk corresponding to the third category of high-risk patients that present with shortness of breath. A collaborative study by Risch and a research group in Brazil [82], found, using multivariate regression analysis, that both obesity and dyspnea pose

Comorbidity CFR from Chinese study [57] Comorbidity Deaths Cases Cardiovascular disease Diabetes Respiratory disease Hypertension Cancer

CFR

92 873 10.5% 80 1102 7.3% 32 511 6.3% 161 2683 6% 6 107 5.6%

Comorbidity CFR from Israeli study [60] Comorbidity Deaths Cases Cardiovascular disease Diabetes Respiratory disease Hypertension Cancer

87 71 23 102 37

CFR

518 16.7% 531 13% 361 6% 744 13.7% 264 10%

FIG. 9: Case fatality rate based on early-stage analysis of COVID19 outbreak in China in the period up to February 11, 2020 [57] vs similar statistics from Israel published on September 7, 2020 [60].

a higher mortality risk than heart disease (see Table 2 of Ref. [82]), therefore, we expect that they both lie in the same 5% to 15% interval as patients with other comorbidities. For the case of obesity, as a mortality risk factor, this conclusion is also supported by more recent meta-analysis [83], showing that obesity is a greater mortality risk factor than diabetes and hypertension, and one that increases with increasing BMI. A study of 148,494 patients across 238 hospitals by the CDC [84] also confirms that obesity is an increasing mortality risk factor with increasing BMI. It is known that obesity is associated with increased levels of the inflammatory cytokines TNF-α (tumor necrosis factor alpha), IL-1 β (interleukin-1beta), and IL-6 (interleukin 6), produced by macrophages in the adipose tissue [85]. A study of 9390 hospitalized patients in Abu Dhabi, United Arab Emirates, has found that patients with severe COVID-19 symptoms, requiring intensive care, had significantly elevated IL-6 biomarker relative to patients that presented with mild or moderate symptoms [86]. An earlier meta-analysis [87] has also confirmed that the IL-6 biomarker is associated with severe progression of the COVID-19 disease.

Consequently, there is a very compelling biological mechanism that explains why obesity is a severe risk factor for progression of the disease to the covid19 pneumonia phase, requiring a high risk classification and immediate early outpatient treatment. For the case of patients presenting with shortness of breath, it is important to appreciate the fact that, without an early outpatient treatment intervention, such presentation implies that the disease is progressing beyond the viral replication phase, into the COVID-19 pneumonia phase, soon to be followed with the thromboembolic stage, oxygen desaturation, and hospitalization. It is thus self-evident that these patients should be classified as high-risk and treated immediately. Assuming that most of such patients will be hospitalized without outpatient treatment, we can also estimate the corresponding mortality

15 risk, in the absence of outpatient treatment, by looking at the conditional probability of death, assuming hospitalization has already taken place. A study by the Houston Methodist Hospital [88] has shown an average mortality rate of 5.8% for hospitalized patients between March 2020 and July 2020, in spite of the use of hydroxychloroquine and anticoagulants. Furthermore, the study reports 12.1% mortality rate, for hospitalized patients between March 13th 2020 and May 15th 2020, and 3.5% mortality rate between May 16th 2020 and July 7, 2020, corresponding to two consecutive surges, noting that the second surge targeted younger patients than the first surge. A prospective multicenter study [89] from Italy of 1050 patients in the Coracle registry, between February 22, 2020 and April 1, 2020, showed an overall 13% average mortality rate, and more specifically, 7.4% mortality rate for hospitalized patients that do not require supplemental oxygen or invasive ventilation, 12.8% mortality rate for hospitalized patients that require supplemental oxygen, and 22.9% mortality rate for hospitalized patients that are invasively ventilated. Based on the above arguments, we can lower bound the untreated mortality risk by 3.5% for the age ≥ 60 demographic and by 5.0% for the high risk patients with comorbidities, obesity, or shortness of breath presentation.

For the age ≥ 50 demographics, we have an expected 2.26% mortality rate for the United States demographic distribution, as estimated by the CDC. The common lower bound for high risk patients in all three categories of the Zelenko case series is thus estimated as 3.5% and it exceeds the efficacy thresholds for both the Zelenko April 2020 [16] and the Zelenko June 2020 [17] case series, and exceeds the selection bias threshold for the Zelenko June 2020 case series [17]. The same untreated mortality rate, lower bound of 3.5% applies to the Raoult case series, which exceeds the efficacy threshold 0.79% and the selection bias threshold 1.40% by a wide margin (factors of 3.8 and 2 respectively). Finally, using the CDC mortality rate of 2.26%, which includes a minority of treated patients and a majority of untreated patients for the age ≥ 50 demographic in the United States as a conservative untreated mortality rate lower bound for the Procter case series, we find that it exceeds the efficacy threshold for both Proctor I [18] and Proctor II [19] case series, and also exceeds the selection bias threshold for the Proctor II case series [19]. A completely different approach is to compare the efficacy and selection bias thresholds against the CFR for the entire population [90]. The CFR for the United States and France is displayed on Fig.

10 for the time period between April 2020 and October 2021. During 2020, the CFR ranged from 2% to 6% in the United States and from 2% to 16% in France. In both countries, the CFR converged to 1.7% during 2021 and remained roughly constant, with very small oscillations throughout 2021. The minimum value of 1.7% exceeds the mortality rate reduction efficacy thresholds for the Zelenko June 2020 [17], Procter II [19], and Raoult case series [20]. It also exceeds the selection bias threshold for the Raoult case series [20]. Using 2.0% as the minimum CFR during 2020, we note that it exceeds the selection bias threshold for the Procter II case series [19] and it equals the selection bias threshold for the Zelenko June 2020 case series [17]. Taking the CFR at

FIG. 10: Cumulative case fatality rate in the United States and France between April 2020 and November 2021.

face value, this is a very strong signal of efficacy, because the CFR includes asymptomatic, low-risk, and high-risk patients, regardless of whether they received early treatment, against solely high-risk patients in the treatment groups of the respective case series. This comparison strongly biases against being able to reject the null hypothesis, and nevertheless, we are still able to do so. In particular, we note that in the United States, the CFR ranged from 2% to 6% during 2020, which lies above the 1.8% mortality rate reduction efficacy threshold for Zelenko April 2020 case series [16]. This was a strong indicator in favor of adopting Zelenko’s triple-drug protocol at that time on an emergency basis, but was unfortunately not implemented in the United States for outpatients [91]. By June 2020, the respective efficacy threshold decreased to 1.0%, and the selection bias threshold decreased to 2.0% while the CFR was still in the neighborhood of 3.0%. Thus, there was a very strong signal in favor of adopting the Zelenko triple drug therapy by the summer of 2020.

4.4.

Analysis of hospitalization rate reduction efficacy

In Fig. 6, we see that the 95% efficacy thresholds for hospitalization rate reduction range from 2.7% to 4.1% for all case series, with the exception of the DSZ case series, where it is at 7.0%, due to the smaller sample size. Likewise, the selection bias thresholds for hospitalization rate reduction with 95% confidence range from 4.2% to 7.3% for all case series, except for the DSZ case series [2]. These thresholds can be compared against the following empirical data. At the beginning of the pandemic, based on data from China until February 11, 2020, there was an initial estimate [59] that the probability of hospitalization for a highrisk age ≥ 60 cohort would range from 10% to 18%. The control group from Zelenko’s study [2] consisting of both low and high-risk patients, again at the beginning of the pandemic here

16 in the United States, reported 377 patients with 58 hospitalizations, corresponding to 15% hospitalization rate. In the Cleveland study [92], which was used to train a predictive model for the risk of hospitalization and death based on patient medical history, the entire dataset consisted of a total of 4,536 patients between March 8, 2020 and June 5, 2020. There were 582 hospitalizations corresponding to 21% hospitalization rate. In the Mass General Brigham hospital study [93], from a cohort of 12,347 patients that tested positive, there were 3,401 hospitalizations between March 4, 2020 and July 14, 2020, corresponding to a 27% hospitalization rate. This was also a cohort that included both low-risk and high-risk patients. The CDC adjusted data [61–63] between February 2020 and September 2021, estimate 13.79% (95% CI: 17.09% to 28.52%) hospitalization probability for the age ≥ 50 group, given a symptomatic infection. For the age ≥ 65 cohort, this estimate increases to 22.09% (95% CI: 17.09% to 28.52%) Overall, our observation is that we tend to see numbers ranging from 10% to 28% with substantial variability between various cohorts, all of which were not given early outpatient treatment. On the other hand, we see that the case series of high risk patients shown in Fig.

6, have efficacy thresholds for hospitalization rate reduction ranging from 2.7% to 4.1%, which have a substantial separation from the 10% to 28% interval. Most remarkably, the hospitalization rate reduction selection bias thresholds also have a substantial separation from the 10% to 28% interval. We interpret this big gap between the two intervals as strong evidence of the existence of hospitalization rate reduction efficacy as a result of the respective early outpatient treatment protocols in the Zelenko April 2020 [16], Zelenko June 2020 [17], Procter I [18], Procter II case series [19]

4.5.

Bayesian analysis of efficacy thresholds

We shall now assess whether the efficacy thresholds need to be increased, using the Bayesian technique described in Section 3, in order to control the false positive rate. In Fig. 11, we have calculated the logarithmic Bayesian metric b(x 0, p2 ), given by Eq. (25), for the mortality and hospitalization rate reduction efficacy thresholds corresponding to 95% confidence, using a range of values of p2 for the purpose of sensitivity analysis. The calculation details are available in our supplementary material document [25]. Recall from Section 3, that p2 corresponds to our sense of the worst possible probability of the respective adverse outcome (hospitalization or death) in high-risk patients in the absence of early outpatient treatment. As such, 5% to 10% is a typical range for mortality rates in untreated high-risk patients, making p2 = 5% a highly conservative choice. We did not consider values higher than 10%, even though worse probabilities are possible, because for p2 > 10%, we see that all logarithmic Bayesian factors already satisfy b(x 0, p2 ) ≥ 2. We have also looked at p2 = 2%, which is obviously entirely unrealistic, because it corresponds to the mortality rate of the Raoult control group [20] where some partial treatment was given. Likewise, for the hospitalization rate reduction efficacy thresholds, we have used the

Bayes factors at the mortality rate efficacy thresholds Study

95% threshold p2 = 0.02 p2 = 0.05 p2 = 0.1

DSZ study 3.8% Zelenko 04/2020 1.8% Zelenko 06/2020 1.0% Procter I 1.7% Procter II 0.84% Raoult 0.79%

N/A 1.17 2.06 1.28 1.92 1.91

1.38 2.04 2.66 2.07 2.48 2.45

1.99 2.45 3.02 2.47 2.82 2.79

Bayes factors at the hospitalization rate efficacy thresholds Study

95% threshold p2 = 0.10 p2 = 0.15 p2 = 0.20

DSZ study 7.0% Zelenko 04/2020 3.2% Zelenko 06/2020 2.7% Procter I 4.1% Procter II 3.6%

1.30 2.00 2.24 1.89 1.98

1.71 2.24 2.47 2.15 2.23

1.92 2.39 2.61 2.32 2.39

FIG. 11: Bayes factor (decimal logarithm) corresponding to the 95% efficacy threshold (Sterne interval [71]) for mortality and hospitalization rate reduction, using maximum untreated mortality rate p2 for high risk patients at p2 ∈ {0.02, 0.05, 0.10} and maximum untreated hospitalization rate p2 for high risk patients at p2 ∈ {0.10, 0.15, 0.20}, for the DSZ study treatment group [2], Zelenko’s complete April 2020 data set [16], Zelenko’s complete June 2020 data set [17], Procter’s observational studies [18, 19], and Raoult’s high risk (older than 60) treatment group [20].

values p2 = 10%, 15%, 20% based on our expectation of a typical 10% to 28% range for the probability of hospitalization, in the absence of early outpatient treatment. We did not consider p2 > 20% since almost all of the logarithmic Bayesian factors satisfy b(x 0, p2 ) ≥ 2 at p2 = 20%. In Fig. 12, we compare the efficacy thresholds for rejecting the null hypothesis with the corresponding 95% confidence Bayesian thresholds, obtained by the inequality b(x 0, p2 ) ≥ 2 for accepting the alternate hypothesis. For the DSZ study [2], we see that the corresponding Bayesian thresholds for hospitalization rate reduction range from 7.2% to 9.5%, which lie above the 7.0% threshold obtained via the p-value. So, the most cautious course of action is to opt for the 9.5% threshold, which is still below most of our estimates for hospitalization probability of untreated patients. For the DSZ study [2], for both p2 = 2% and p2 = 5%, the logarithmic Bayesian factor for mortality rate reduction does not go above the decisive threshold for any value of x with a/N ≤ x ≤ p2 , consequently the corresponding Bayesian thresholds are undefined, and for p2 = 10% we find a Bayesian mortality rate reduction threshold of 3.9% which is slightly larger than the p-value threshold of 3.8%.

For the Procter I case series [18], there is a weak indication that the 4.1% efficacy threshold for hospitalization rate reduction might have to be increased to 4.3%, and the mortality rate reduction threshold increased from 1.7% to 1.9%. Likewise for the Procter II case series [19], an increase of the hospitalization rate reduction efficacy threshold from 3.6% to 3.7% is weakly indicated. Both adjustments are negligible and inconsequential. For the Zelenko April 2020 [16] and Ze-

17 Mortality rate Bayesian efficacy thresholds

Bayes factors at the mortality rate efficacy thresholds

Study

Study

95% log Bayes = 2 thresholds threshold p2 = 2% p2 = 5% p2 = 10%

DSZ study 3.8% Zelenko 04/2020 1.8% Zelenko 06/2020 1.0% Procter I 1.7% Procter II 0.84% Raoult 0.79%

N/A N/A 1.0% N/A 0.87% 0.82%

N/A 1.8% 0.8% 1.9% 0.7% < 0.7%

3.9% 1.5% 0.6% 1.3% 0.6% < 0.7%

99% threshold p2 = 0.02 p2 = 0.05 p2 = 0.1

DSZ study 5.3% Zelenko 04/2020 2.4% Zelenko 06/2020 1.2% Procter I 2.3% Procter II 1.08% Raoult 0.96%

Study

DSZ study 8.8% Zelenko 04/2020 3.9% Zelenko 06/2020 3.0% Procter I 4.9% Procter II 4.0%

9.5% 3.2% 2.6% 4.3% 3.7%

7.8% 3.0% 2.5% 4.0% 3.5%

7.2% 2.9% 2.4% 3.7% 3.4%

FIG. 12: Comparison of the 95% confidence efficacy threshold (Sterne interval [71]) for mortality and hospitalization rate reduction with the Bayes factor efficacy thresholds at log Bayes = 2, using maximum untreated mortality rate p2 for high risk patients at p2 ∈ {0.02, 0.05, 0.10} and maximum untreated hospitalization rate p2 for high risk patients at p2 ∈ {0.10, 0.15, 0.20}, for the DSZ study treatment group [2], Zelenko’s complete April 2020 data set [16], Zelenko’s complete June 2020 data set [17], Procter’s observational studies [18, 19], and Raoult’s high risk (older than 60) treatment group [20].

lenko June 2020 [17] case series, where the sample sizes are much larger, we see that the overall trend is for the Bayesian thresholds to be far more lenient than the ones obtained via the p-value. This is possibly attributed to a very strong signal of efficacy in the data. It is interesting to repeat the Bayesian analysis on the efficacy thresholds for mortality rate reduction and hospitalization rate reduction for 99% confidence and 99.9% confidence. We have seen that the Bayesian adjustments to the 95% confidence efficacy thresholds, when they are needed, are very small, so the relevant question is whether this pattern continues when the demanded confidence increases to 99% or 99.9%. Fig. 13 and Fig. 14 show the values of the logarithmic Bayesian factor b2 (x 0, p2 ) at the mortality and hospitalization efficacy thresholds for 99% and 99.9% confidence, as determined solely from the p-value, and for various values of p2 , as previously discussed. Note that for Fig. 13 the decisive Bayesian factor threshold corresponding to 99% confidence is b2 (x 0, p2 ) ≥ 2.7. Likewise, in Fig. 14, the decisive Bayesian factor threshold corresponding to 99.9% confidence is b2 (x 0, p2 ) ≥ 3.7. We see that for the most part the logarithmic Bayesian factors are either above or near their respective thresholds. Likewise, in Fig. 15 and Fig.

16 we are comparing the mortality and hospitalization rate reduction efficacy thresholds determined via the p-value, against the corresponding efficacy

2.70 3.27 3.57 3.17 3.53 3.51

99% threshold p2 = 0.10 p2 = 0.15 p2 = 0.20

Study

DSZ study 7.0% Zelenko 04/2020 3.2% Zelenko 06/2020 2.7% Procter I 4.1% Procter II 3.6%

N/A 2.81 3.21 2.72 3.17 3.16

Bayes factors at the hospitalization rate efficacy thresholds

Hospitalization rate Bayesian efficacy thresholds 95% log Bayes = 2 thresholds threshold p2 = 10% p2 = 15% p2 = 20%

N/A N/A 2.53 N/A 2.55 2.57

1.83 2.75 2.77 2.55 2.63

2.42 3.00 3.00 2.85 2.89

2.67 3.17 3.16 3.02 3.05

FIG. 13: Bayes factor (decimal logarithm) corresponding to the 99% efficacy threshold (Sterne interval [71]) for mortality and hospitalization rate reduction, using maximum untreated mortality rate p2 for high risk patients at p2 ∈ {0.02, 0.05, 0.10} and maximum untreated hospitalization rate p2 for high risk patients at p2 ∈ {0.10, 0.15, 0.20}, for the DSZ study treatment group [2], Zelenko’s complete April 2020 data set [16], Zelenko’s complete June 2020 data set [17], Procter’s observational studies [18, 19], and Raoult’s high risk (older than 60) treatment group [20].

thresholds determined using the logarithmic Bayesian factor b2 (x 0, p2 ), for 99% and 99.9% confidence correspondingly. We see that the Bayesian perturbations to the efficacy thresholds are for the most part negligible for both 99% and 99.9% confidence, continuing the similar pattern that we have observed for the 95% confidence efficacy thresholds. Based on these results we conclude that for the case series under consideration, we see that the Bayesian adjustments to the efficacy thresholds for mortality and hospitalization rate reduction are negligible and therefore do not impact the analysis of the preceding sections.

5.

DISCUSSION AND CONCLUSIONS

Our findings fully support risk stratification in the management of acute COVID-19, with the intent of reducing the intensity and duration of symptoms and by that mechanism, lower the risk of hospitalization and death. Although COVID19 is generally known as a respiratory disease, there is an accumulation of evidence [40, 94, 95] that it is also, if not primarily, a vascular disease, with endothelial injury having a major role in sustained permanent injuries, hospitalizations, and death. The spike protein has been shown to damage the vascular endothelial cells [40] by downregulating ACE2, thereby inhibiting mitochondrial function, and by impairing the bioavailability of nitric oxide to endothelial cells. The spike protein also triggers immune dysregulation, triggering endothelial cells to transition to an activated immune response

18 Mortality rate Bayesian efficacy thresholds

Bayes factors at the mortality rate efficacy thresholds Study

99.9% threshold p2 = 0.02 p2 = 0.05 p2 = 0.1

DSZ study 7.0% Zelenko 04/2020 2.9% Zelenko 06/2020 1.6% Procter I 3.1% Procter II 1.4% Raoult 1.18%

N/A N/A 3.43 N/A 3.38 3.49

N/A 3.47 4.34 3.59 4.15 4.16

3.51 4.00 4.73 4.16 4.53 4.52

Bayes factors at the hospitalization rate efficacy thresholds Study

99.9% threshold p2 = 0.10 p2 = 0.15 p2 = 0.20

DSZ study 10.6% Zelenko 04/2020 4.7% Zelenko 06/2020 3.5% Procter I 5.9% Procter II 4.5%

N/A 3.68 3.75 3.45 3.54

3.17 3.97 4.00 3.80 3.82

3.49 4.15 4.16 3.99 3.99

FIG. 14: Bayes factor (decimal logarithm) corresponding to the 99.9% efficacy threshold (Sterne interval [71]) for mortality and hospitalization rate reduction, using maximum untreated mortality rate p2 for high risk patients at p2 ∈ {0.02, 0.05, 0.10} and maximum untreated hospitalization rate p2 for high risk patients at p2 ∈ {0.10, 0.15, 0.20}, for the DSZ study treatment group [2], Zelenko’s complete April 2020 data set [16], Zelenko’s complete June 2020 data set [17], Procter’s observational studies [18, 19], and Raoult’s high risk (older than 60) treatment group [20].

state, which causes both microvascular and diffuse microvascular thrombosis, leading to myocardial injury and other organ damage [94, 95]. The most impactful countermeasure that can prevent these adverse outcomes is early outpatient treatment, using multiple drugs in combination, that stops viral replication at the first phase of the illness, and mitigates the injuries caused by the hyper inflammatory COVID-19 pneumonia phase and the subsequent thromboembolic phase. Looking back, after 2 years of going through the COVID19 pandemic, one of the lessons learned is that some of the key discoveries for the successful treatment of a novel disease emerge from the experience of the frontline doctors that are directly confronted with the need to find a way to help their patients. Although the orthodox approach is to consider possible treatments as unproven until they are validated with an RCT, in real life, it is possible to be confronted with a situation where the observational data is sufficiently strong to justify the immediate adoption of a treatment protocol, and to raise the ethical concern of whether it is appropriate to even conduct the RCT, and deny treatment to a very large cohort of patients, in order to form a control group. Consequently, there is a need to be able to analyze observational data in a statistically rigorous way.

We have provided a hybrid statistical framework for assessing observational evidence that combines both frequentist and Bayesian methods; the frequentist methods aim to control the p-value for rejecting the null hypothesis, whereas the Bayesian methods aim to control the false positive rate. The two methods are complementary and not mutually exclusive.

Study

99% log Bayes = 2.7 thresholds threshold p2 = 2% p2 = 5% p2 = 10%

DSZ study 5.3% Zelenko 04/2020 2.4% Zelenko 06/2020 1.2% Procter I 2.3% Procter II 1.08% Raoult 0.96%

N/A N/A 1.3% N/A 1.14% 1.0%

N/A 2.4% 1.1% 2.3% 0.92% 0.86%

5.3% 2.0% 0.9% 1.9% 0.80% 0.77%

Hospitalization rate Bayesian efficacy thresholds Study

99% log Bayes = 2.7 thresholds threshold p2 = 10% p2 = 15% p2 = 20%

DSZ study 8.8% Zelenko 04/2020 3.9% Zelenko 06/2020 3.0% Procter I 4.9% Procter II 4.0%

N/A N/A 3.0% 5.1% 4.1%

9.5% 3.7% 2.9% 4.8% 3.9%

8.9% 3.5% 2.8% 4.6% 3.8%

FIG. 15: Comparison of the 99% confidence efficacy threshold (Sterne interval [71]) for mortality and hospitalization rate reduction with the Bayes factor efficacy thresholds at log Bayes = 2.7, using maximum untreated mortality rate p2 for high risk patients at p2 ∈ {0.02, 0.05, 0.10} and maximum untreated hospitalization rate p2 for high risk patients at p2 ∈ {0.10, 0.15, 0.20}, for the DSZ study treatment group [2], Zelenko’s complete April 2020 data set [16], Zelenko’s complete June 2020 data set [17], Procter’s observational studies [18, 19], and Raoult’s high risk (older than 60) treatment group [20].

The core mathematical ideas are very simple, however, there are some subtleties concerning the relationship of the frequentist methods with the exact Fisher test and the binomial proportion confidence interval problem. The main weakness of the proposed statistical methodology is that it has to be limited only to the assessment of treatments that are based on repurposed medications [66] with a known excellent safety record. It would be highly inappropriate to use this approach on new medications, or other countermeasures, where the balance of risks and benefits is yet to be determined. Furthermore, the analysis of the treatment group case series needs to be compared with a model that can at minimum lower-bound the probability of adverse outcomes without treatment, based on our prior knowledge. On the other hand, the development of this model can be done independently from the analysis of the treatment group case series. One way in which our approach deviates from the usual way of doing things, is that we are using the proposed statistical methodology to assess the efficacy of the entire treatment algorithm against supportive care. Both of the original Zelenko protocol [2] and the more enhanced McCullough protocol [21–23] are examples of sequenced multi-drug treatment protocols.

Furthermore, both protocols are algorithmic, in the sense that treatment is customized to the individual patient, based on the patient’s medical history and the response to treatment. For the case of the Zelenko protocol [2] this

19 Mortality rate Bayesian efficacy thresholds Study

99.9% log Bayes = 3.7 thresholds threshold p2 = 2% p2 = 5% p2 = 10%

DSZ study 7.0% Zelenko 04/2020 2.9% Zelenko 06/2020 1.6% Procter I 3.1% Procter II 1.4% Raoult 1.18%

N/A N/A 1.8% N/A 1.53% 1.23%

N/A 3.1% 1.4% 3.2% 1.26% 1.08%

7.4% 2.7% 1.3% 2.8% 1.14% 1.01%

Hospitalization rate Bayesian efficacy thresholds Study

99.9% log Bayes = 3.7 thresholds threshold p2 = 10% p2 = 15% p2 = 20%

DSZ study 10.6% Zelenko 04/2020 4.7% Zelenko 06/2020 3.5% Procter I 5.9% Procter II 4.5%

N/A 4.8% 3.5% 6.2% 4.6%

11.9% 4.5% 3.4% 5.8% 4.5%

11.1% 4.4% 3.3% 5.7% 4.4%

FIG. 16: Comparison of the 99.9% confidence efficacy threshold (Sterne interval [71]) for mortality and hospitalization rate reduction with the Bayes factor efficacy thresholds at log Bayes = 3.7, using maximum untreated mortality rate p2 for high risk patients at p2 ∈ {0.02, 0.05, 0.10} and maximum untreated hospitalization rate p2 for high risk patients at p2 ∈ {0.10, 0.15, 0.20}, for the DSZ study treatment group [2], Zelenko’s complete April 2020 data set [16], Zelenko’s complete June 2020 data set [17], Procter’s observational studies [18, 19], and Raoult’s high risk (older than 60) treatment group [20].

is done via the risk stratification of patients to low-risk and high-risk patients. For the case of the McCullough protocol [21–23], this is done both by risk stratification and also by accounting for the progression of the illness through the three distinct stages and response to treatment. Consequently, the immediate goal is not to establish that any one particular drug is effective. The goal is to establish that the treatment algorithm itself is effective, so that it can be deployed rapidly on an emergency basis and be subsequently improved over time with further research. A possible theoretical criticism is that the particular case series that we have considered may have selection bias, i.e. they were published and came to our attention as a result of their positive outcomes, and not because they are representative of the typical outcomes of most treatment centers that employ early outpatient treatment protocols against COVID19. This is mitigated to some extent by the fact that we have reported case series from three different treatment centers, two in the United States and one in France, with consistent mortality rates, therefore this consistency is compelling statistical evidence against selection bias.

More importantly, for both of the Zelenko [2, 16, 17] and Procter [18, 19] case series, we have two consecutive reports over two consecutive time intervals replicating the hospitalization and mortality rate reduction outcomes, and these replications are additional statistical evidence against a theoretical selection bias. Finally, we

have introduced the idea of selection bias thresholds that can be used to account for random selection bias. For the Zelenko June 2020 [17], Proctor II [19] and Raoult [20] case series, we can have 95% confidence that random selection bias cannot be entirely responsible for the positive signal of benefit in mortality and hospitalization rate reduction. The case series that we have analyzed in this paper add up to a total of 3164 high-risk patients. It is currently estimated that the total number of high-risk patients that have been treated with early outpatient treatment protocols throughout the United States may exceed this number by one or two orders of magnitude [51]. Unfortunately, no resources have been allocated to study this data by our public health agencies, but we can make some suggestions about how such an analysis could be carried out. One idea for quickly analyzing a very large dataset is to extract the age > 50 and/or age > 65 part of the database, calculate the corresponding efficacy thresholds for hospitalization rate reduction and mortality rate reduction, and compare them with the CDC estimates [61–63] for number of hospitalizations and deaths for these age groups over the total number of cases with symptomatic illness.

Given a large enough data set, it would also be interesting to risk-stratify the age > 50 and/or age > 65 cohorts further with respect to number of days between initial symptoms and initiation of treatment and calculate the efficacy thresholds as a function of the delay in initiating treatment. Furthermore, it would be useful to break down the case series data in sequential time intervals corresponding to different waves and different variants of the SARS-CoV-2 virus. This analysis would inadvertently not include younger patients that are high risk due to comorbidities or shortness of breath presentation, however, it has the advantage that it can be carried out quickly with limited resources. Analyzing the data from several more treatment centers, that have adopted early outpatient treatment protocols for high-risk patients would further mitigate the potential for selection bias. With substantial resources, a more detailed analysis is possible that can consider the entire dataset and actually estimate the treatment efficacy. Given a case series of N patients, one can input the medical history of each patient to the Cleveland Clinic calculator [92] and use their mathematical model to predict the probability of hospitalization and death for each patient individually. Knowing the corresponding sequence of probabilities q = (p1, p2, . . .

, p N ) for an adverse outcome (hospitalization or death) for all patients, the probability pr(N, a|q) of seeing a adverse outcomes follows a Poison binomial distribution [96], and it can be substituted to Eq. (2) in order to calculate the p-value for rejecting the null hypothesis of no treatment efficacy. Because the probability of an adverse outcome is known for each patient, note that there is no need to worry about calculating any efficacy thresholds, and it is possible instead to directly calculate the p-value for rejecting the null hypothesis.. Furthermore, since the mean of the Poison binomial distribution is the average q = (1/N )(p1 + p2 + . . . + pn ) of the individual probabilities, one can calculate the risk ratio via the equation RR = a/(qN ). To conduct the corresponding Bayesian analysis, we can assume that the effect of the early outpatient treatment is to re-

20 duce the probabilities of adverse outcome by a numerical factor x to xq = (xp1, xp2, . . . , xp N ) with 0 ≤ x ≤ 1 and use the Poison binomial distribution pr(N, a|xq) in Eq. (28) and Eq. (31) to calculate the corresponding integrals needed for the Bayesian factor. All other aspects of the Bayesian analysis would remain the same, except that the hypothesis being validated would not concern any efficacy thresholds but it would instead concern hypotheses about the actual efficacy x of the early outpatient treatment protocol. That said, we do not mean to imply that such a detailed analysis is necessary in order to greenlight the use of the investigated early outpatient treatment protocols for COVID19. However, the fact that such a detailed analysis is possible to carry out, using existing data and prior mathematical modeling, is highly relevant with respect to assessing the ethics of validating the McCullough protocol at this time, using an RCT. A limitation of the Cleveland Clinic calculator is that it should ideally be used in conjunction with case series over time intervals that are aligned with the data set used to train the calculator’s mathematical predictive model. Because the Cleveland Clinic calculator used data collected between March 4th 2020 and July 14th 2020 it can certainly be applied to case series up until July 2020.

However, we believe that it can also be extended up until and including the Delta variant, that became dominant towards the end of 2021, since all of these subsequent variants were just as hard to treat or harder than the initial waves in 2020. Notwithstanding economically-motivated obstacles [91, 97] that have been placed against the adoption of early treatment protocols for COVID-19, everything that we have been through during the last two years vindicates the position of Frieden [53] that there is an urgent need to leverage and overcome the limitations of real-world evidence data, in order to deploy a timely life-saving response to urgent health issues. There is still an opportunity to learn much by analyzing data from various treatment centers here in the United States that treated COVID-19 with early outpatient treatment protocols, as well as treatment centers from all around the world. More importantly, it is necessary to reflect on and develop policies and procedures for leveraging the direct experience of frontline doctors treating patients, towards an agile and effective response to future epidemics and pandemics. With this paper, we have put forth a statistical methodology that can be used to assess the strength of the evidence accumulated by this direct experience. However, several other challenges remain unresolved.

The regulatory capture by pharmaceutical corporations of public health institutions in some nations, including research funding agencies, has biased them against drug repurposing [91, 97]. It is therefore important to discover ways to reverse that and to also introduce decentralization of authority and failsafes to mitigate the impact of failure of any one particular institution. More specifically, the scope of public health authorities has to be limited to providing data and logistical support, but should not be allowed to interfere in the doctor-patient relationship. A substantial percentage of medical doctors are employed by hospitals and health systems or are restricted by insurance contracts. We have seen a need to restore the authority of doctors to prescribe off-label medica-

tions without harassment by pharmacies, medical boards, hospital administrators, or insurance companies [98, 99]. There is also a need for channels of communication free of censorship, so that doctors can rapidly communicate their findings with each other and with the public. Additional channels of communication are needed between doctors and public health agencies at the state and federal levels. Finally, there are technical logistics involved in processing and analyzing the medical records that form the respective treatment case series. Lack of resources is the main reason why extensive currently available data, accumulated during the last two years have not yet been analyzed. Solutions to these challenges could be pursued at the federal and/or state level and perhaps also via professional associations at the grassroots level.

Author contributions

EG conceptualized the mathematical framework, analyzed the case series data, and wrote the first draft of the paper. PAC and VZ contributed to the literature review, as well as writing and editorial changes to the manuscript. VZ contributed copies of his three public letters [16, 17, 24], attached to our supplementary material document [25], containing some of the consecutive case series data analyzed in the manuscript. VZ also contributed data on how his treatment protocol evolved over time during 2020. All authors read and approved the final version of the manuscript.

Ethics approval and consent to participate

Not applicable.

Acknowledgements

It is a pleasure to thank Roland Derwand, Harvey Risch, and Marc Rendell for correspondence and encouragement. In particular, we wish to thank Roland Derwand for bringing the Israeli study [60] to our attention, and Harvey Risch for highlighting the importance of the papers by Frieden [53] and Deaton and Cartwright [54] and for the suggestion that we look into the coverage probability, at an early stage of our work. We also thank Marc Rendell for suggesting that we introduce a quantitative technique to mitigate selection bias, and the anonymous reviewers for very helpful suggestions that have gone a long way towards improving the manuscript. Finally, wish to acknowledge Lawrence Huntoon for encouraging us to look into a Bayesian approach, at a very early stage of this research project.

Funding

The research has received no external funding.

21 Conflict of interest

The authors declare no conflicts of interest. Peter A. McCullough is serving as the editor-in-chief of this journal. Appendix A: Exact Fisher test in the limit of an infinite control group

Let N be the total number of patients in the treatment group, let a be the number of patients with an adverse outcome (hospitalization or death) in the treatment group, let M be the total number of patients in the control group, and let b be

pr(N, a, M, b) =

a+b b

!

the number of patients in the control group with an adverse outcome. In this appendix we will show that in the limit of an infinite control group (M, b) with x = b/M, the p-value p(N, a, M, b) obtained from the two-tail exact Fisher test converges to p(N, a, x). In the exact Fisher test, we assume that N, M, and a + b, are fixed numbers, and under the null hypothesis, we also assume that the distribution of the total a + b patients with an adverse outcome between the treatment group and control group is random, with equal probability for every possible combination. It follows that under the null hypothesis, the probability of seeing a particular event (N, a, M, b) is given by

! N+M−a−b N−a ! . N+M N

(A1)

The corresponding p-value is the probability of observing the event (N, a, M, b) or any other less probable event, and it is given by p(N, a, M, b) =

min{N,a+b X }

pr(N, n, M, a + b − n)H (pr(N, a, M, b) − pr(N, n, M, a + b − n)),

(A2)

n=0

We note that the summation variable n is restricted by both the total size N of the treatment group and the total number a + b of the patients with an adverse outcome, so the permissible range for all possible events is 0 ≤ n ≤ min{N, a + b}. A key insight is that in the definition of pr(N, a, M, b), the variable M can be replaced with a continuous real number, because it appears only in the top argument of the corresponding binomial coefficients. Recall that for all a ∈ R and n ∈ N the extended definition of the binomial coefficient is given by ! n n n a 1 Y 1 Y 1 Y = (a + 1 − λ) = (a + 1 − (n − λ + 1)) = (a − n + λ). (A3) n n! λ=1 n! λ=1 n! λ=1 On the second step we have used the transformation λ 7→ n − λ + 1 which effectively reverses the order of factors in the product. It follows that for all M ∈ R the corresponding M-dependent binomial coefficients are given by ! N −a N −a Y Y N+M−a−b 1 1 = ((N + M − a − b) − (N − a) + λ) = (M − b + λ), N−a (N − a)! λ=1 (N − a)! λ=1

(A4)

and ! N N N+M 1 Y 1 Y = ((N + M) − N + λ) = (M + λ), N! λ=1 N! λ=1 N and thus, the hypergeometric probability distribution pr(N, a, M, b) can be rewritten as ! ! a+b N+M−a−b b N−a ! pr(N, a, M, b) = N+M N !    Y N −a N Y  (a + b)!  1 1   = (M − b + λ)   N!  a!b!  (N − a)! λ=1 M + λ      λ=1 

(A5)

(A6)

(A7)

22 ! N −a N Y (a + b)! Y 1 N! (M − b + λ) = a!(N − a)! b! M+λ λ=1 λ=1 !Y ! NY ! a N −a a −a Y Y 1 N 1 (b + λ) (M − b + λ) = a λ=1 M + λ λ=1 M + a + gl λ=1 λ=1 ! a ! N −a ! N Y b+λ Y M −b+λ = . a λ=1 M + λ λ=1 M + a + λ

(A8)

(A9)

(A10)

To take the limit of an infinite control group with probability x of an adverse outcome, we set b = x M, or equivalently M = (1/x)b, and take a sequence limit b ∈ N to infinity. We conclude that ! Y !   NY !  a   −a b+λ (1/x)b − b + λ  lim lim    b ∈N (1/x)b + λ   b ∈N (1/x)b + a + λ   λ=1   λ=1  ! !a ! N −a 1 1/x − 1 N = a 1/x 1/x ! N a = x (1 − x) N −a = pr(N, a|x). a

lim pr(N, a, (1/x)b, b) =

b ∈N

N a

An immediate consequence is that the corresponding p-values satisfy a similar relationship that reads lim p(N, a, (1/x)b, b) = p(N, a|x).

b ∈N∗

(A14)

The probability sums on both sides of Eq. (A14) involve a variable n that goes from 0 to N, making the number of terms on the left-hand-side probability sum independent of the size of the control group, as soon as b is large enough. This makes it possible to derive Eq. (A14) as an immediate consequence of Eq. (A13).

Appendix B: Calculation of the selection bias thresholds

We provide the mathematical justification for the calculation of the selection bias threshold given by Eq. (7). Suppose that we have a case series (N, a) of N treated patients, with a adverse outcomes, and have already calculated an appropriate efficacy threshold x 0 , using the techniques detailed in Section 2 and Section 3. Let x be the corresponding probability of an adverse event, in the absence of treatment, as has been observed at the population level, under the same selection criteria for high-risk patient classification, as used for forming the treated case series. The true rate x 0 of an adverse event for the selected N patients, had they not received treatment, could range from a minimum value m1 (N, x, p0 )/N to a maximum value m2 (N, x, p0 )/N, and we would like to be able to assert with 1−p0 confidence that m1 (N, x, p0 )/N ≤ x 0 ≤ m2 (N, x, p0 )/N. The problem of determining m1 and m2 is the “mirror image” of the binomial confidence interval problem. With the latter case, we have a known observed event and seek a confidence interval for the probability that generated the event. Here, we have a given probability x and need the confidence interval for