Vasily Vlassov, Olga Rebrova, Valerii Aksenov

Russian Society for Evidence-Based Medicine

February 05, 2021

The authors declare that they have no conflicts of interest regarding any COVID-19 vaccines.

Despite the announcement of the preliminary results of a randomized controlled trial (RCT) of the first Russian vaccine in November 2020 and the promise of a future full publication in an international journal to take place in November, the publication took place only on February 2, 2021 (1). Given the tremendous social significance of this development and decisions regarding citizens of this country, we considered it necessary to publish the following commentary on this article.

It should be recalled that the vaccine received Emergency Use Authorization in Russia after reduced phase 1-2 trials involving an unacceptably small number of healthy young volunteers. The publication of these trial results raised doubts about the study’s quality among specialists in Russia and abroad (2). The published corrections to the article, not accompanied by explanations of the origin of the text’s errors, only exacerbated the distrust of the report (3). That is why the results of the phase 3 RCT have long been expected by Russian society.

It is important to emphasize that the other vaccine developers who received temporary approval for their use simultaneously as the developers of Sputnik-V not only have already released the results of the preliminary analysis of their phase 3 RCTs but have also released the trial protocols. The Russian vaccine developers have limited themselves to press releases from the trial sponsor, and the report we briefly comment on is the only source of information beyond the limited RCT registration data on clinicaltrials.gov.

Only from this article do we learn that the protocol reflected in the RCT summary on clinicaltrials.gov (NCT04530396) was revised on November 5, 2020, which is not reflected in this register. We are only told that at that time, it was decided to conduct three interim data analyzes. This article includes the analysis results as of November 24, 2020.

From Fig. 1 (flow chart), we learn that, for various not entirely clear reasons, 74/16501 (0.45%) patients were excluded from the number of those who received the first dose­—in the vaccine group, and 41/5476 (0.75%) patients, in the placebo group. The number of those excluded at this stage is small and could hardly have influenced the result. However, it should be taken into account that the total numbers of participants planned, screened, included in the study, and this analysis are very different. That is, 40,000 were planned for inclusion in the trial; 35,963 were screened; 33,758 were actually enrolled (reported on clinicaltrials.gov, ‘actual enrollment’); 21,977 were randomized; 21,862 were analyzed for the primary outcome (named as such in the register) and for serious adverse events (SAEs); 19,866 for another outcome (quoted in the article as the primary one); 12,296 for adverse events. Thus, the published results are obtained in selective groups, and there are signs of unjustified exclusion of cases.

The article provides data on the immunogenicity of the vaccine. They are derived from an analysis of blood samples from RCT participants, who can be called a ‘convenience sample.’ The authors provide no evidence that this sample is representative of all RCT participants. Accordingly, we refrain from the discussing these results.

Discussing the RCT limitations, the authors note that cases of COVID-19 were detected by self-observation of patients, assessed by a doctor, and confirmed by PCR. Thus, asymptomatic cases were not detected. It can be assumed that there were more such cases in the vaccinated group than in the placebo group. The result of such a measurement of incidence may be an underestimation of the incidence in the vaccinated group to a greater extent than in the placebo group and thus an overestimation of the protective effect of the vaccine. The size of this bias in the vaccine efficacy evaluation cannot be estimated. The problem with choosing clinically detectable disease as the primary outcome is present not only in this trial but also in trials of other COVID-19 vaccines. This is not only a problem of estimation of accuracy. If vaccination leads to a significant reduction in the disease’s severity, then vaccinated people who have a mild infection or those asymptomatic can be active carriers of the infection. In other words, vaccination may accelerate the infection spread rather than slow it down. In any case, like the RCTs of other vaccines already approved on temporary conditions for use, this RCT did not aim to assess the effect of vaccination on the infection spread.

Cases of common cold classified by doctors as COVID-19 were recognized as such and included in the RCT statistics as the primary outcomes only if confirmed by a PCR test. The article does not indicate which tests were used for these purposes and what their operational characteristics are.

The analysis of the results was carried out with a deviation from the protocol, in the part in which we are familiar with the latter from the report on clinicaltrials.gov. The protocol prescribed an analysis of outcomes within 6 months after the administration of the first dose. When analyzed for the timeframe from the administration of the first dose, the effectiveness of the vaccine is 73.1% (as calculated by the authors of the article). The effectiveness of 91.6% is declared and presented in the article’s conclusion as the main result is obtained if the days before the second dose are excluded from the analysis. There are reasons for this variant of the analysis, but these reasons were known as well during the protocol development. The principal analysis should be performed according to the protocol. Deviation from the protocol during the analysis is acceptable, but the results of such an analysis are usually considered as exploratory ones, requiring additional confirmation in a separate experiment.

In this study, the follow-up period for participants varies greatly. Therefore, only the survival analysis can be considered correct, but its results are not presented in the article. Only Figure 2 is shown. Our data reconstruction shows a statistically significant difference between the groups in the log-rank test (P < 0.001). The measure of the magnitude of the effect, in this case, should be the hazard ratio. The calculation of efficacy given in the article appears to be incorrect.

The vaccine is presented as safe. The presented frequencies indicate approximately the same incidence of SAEs with the administration of the vaccine and placebo. The incidence of SAEs (0.3%) is only 10 times lower than the disease incidence (2%), which cannot be considered unremarkable but should not be overestimated since 2% is the cumulative incidence over the analysis period. Over the year, the number of cases should increase, and an improvement in this ratio can be expected.

34% of cases were excluded from the safety analysis. The authors explain this by delaying obtaining information, which does not seem entirely convincing, given the information technologies used and the importance of the issue.

The data on deaths are insufficient. In particular, the ages of the two patients who died from COVID-19 in the vaccination group were not specified. The authors regard these cases as ones included in the trial while in the incubation period of the disease. In our opinion, such a simple interpretation is insufficient. A much more detailed description of these two cases is required.

The follow-up period for the trial participants is too short — the median is 48 days. With the negligibly small sample size of phase 1-2 trial and the phase 3 RCT actually have been completed, this means that the vaccine’s safety beyond one and a half months remains unknown. Before the present epidemic, adenoviral vector vaccines had not been approved in the world, and an abridged trial of a fundamentally new technology intended for mass use does, indeed, cause concern.

When assessing the risk of bias of the results using the Cochrane Risk-of-Bias tool, we find that due to missing outcome data and the authors having a marked conflict of interest, the overall risk of bias of the study results is high.

The article’s text also contains numerous other errors, ambiguities of the text, possible misprints, which we are not discussing here. For example, in Fig. 2, the number of people observed in the vaccination group on the 20th day is greater than that on the 10th day (15,717 and 15,338). It is impossible, just as it is impossible to become a twenty-year-old without previously having been a ten-year-old.

The authors have fulfilled the journal’s condition and declared their readiness to provide access to the original data. However, this time they have locked out the access to the raw data, setting so many conditions, including the approval by the ‘security department,’ that no one will likely turn to them for the raw data. This is rather unfortunate since the lack of readiness for constructive interaction with scientific communities provides grounds for the most serious suspicions. Up to suspicions of data falsification because of unexplained discrepancies or failure to report essential methods or results.

The authors offer readers an interim report for November 24, creating an illusion of the trial being ongoing and the final report to appear on schedule. Meanwhile, it has been known for at least 50 days that the trial is, in fact, interrupted, the placebo group is partially unmasked. This means that no data on comparison between the vaccinated and unvaccinated groups will be obtained any longer. The next report can only contain information about the disease incidence among the vaccinated. This information will be useful, but it should be understood that this interim report is, in fact, the final report on the phase 3 RCT of the Sputnik-V vaccine.

Conclusion

The feasibility of using any medical intervention is determined by the balance of benefits and harms from its use, i.e., its efficacy and safety ratio. An analysis of the phase 3 trial data reported in the publication and the RCT registry leaves both these aspects of this intervention highly uncertain.

References

1. https://doi.org/10.1016/S0140-6736(21)00234-8

3. https://doi.org/10.1016/S0140-6736(20)32681-7

2. https://trv-science.ru/2020/11/covid-19-logunov-retraction/

Acknowledgement. The authors thank P.V. Zhelnov for participating in the translation of the commentary into English.