Overwhelming efficacy - the stats required, page-573

Ok some posters asked for an update based on the press release, and specifically for a futility check to see what our effectiveness would have been at the 30 and 45% readouts to still suffer futility at the 60%. I would have to rewrite the code in order to do a futility analysis, so I can't deliver there. What I can do though, is just post a few thoughts to close this thread on what the actual trial design was vs what I was modelling.

One huge mistake I made was in assuming the desired improvement %. At no point in time did I add a range of percentages to the uncertainty grid. I just was working off an assumption that I found in a white paper. After finding out they were seeking a 43% reduction in mortality I was really shocked. In my modelling I was using a 10% reduction in mortality and that was fixed on all models. Where I was nervous about the choice of prior, I should have been nervous about the desired improvement percentage. When I run the model with a 43% improvement percentage the chance of getting overwhelming efficacy is very low even with a neutral prior. It's a tough mark to hit.

Given the treatment passed the 30% AND 45% interim analyses, it does seem like it would have been making a difference of some sort, even as SOC improved to save more lives, however there's probably a number of reasons this didn't continue to translate into a large reduction in mortality as the trial progressed, notably our inclusion criteria did not account for age and required ventilation, so as treatments changed and we got more information as the pandemic went on, our actual cohort for treatment would have been very different to the start. If you read back through this thread I posted a white paper about the changing practice in covid treatment to use less ventilation, which means we were starting to get only the worst of the worst patients, so far gone they were beyond saving, as opposed to early in the pandemic when we had a range of patients. This means that control mortality would have stayed high but that we were no longer treating the same class of patients and our treated mortality would have also been high. I would guess that they will likely find out after reviewing the data that the treatment works if administered earlier in the treatment cycle.

Ok that's all for now. If they release the final trial data I might do one last update, but if not then it was a pleasure chatting stats with you all!