HI All,
I thought I would start this thread just to make it easy to find for anyone looking for scenarios by which we get overwhelming efficacy. In this post I'm going to map out what it usually requires, what it will take to achieve it based on what we know about the trial numbers, and some personal thoughts on how likely it is (though the intent is to present impartial information so you can make up your own mind).
I presented a few scenarios on efficacy here:
https://hotcopper.com.au/threads/msb-most-exciting-6-months.5397246/?post_id=46767990This is good background reading if you are bored.
In this post and thread we'll be talking a lot about p values, if you've never heard of these before, basically a p value is how significant a result is. the normal standard for significance (scientifically proven) is p < 0.05 which you could also express as "95% sure it works" though I hard heard medical trials demand a p value of p < 0.01 which would be "99% sure it works" (can anyone confirm this?). Scientists usually design trials to achieve p values way better than p < 0.01 so that there's less risk of failure. This usually involves a longer trial with more subjects and controls to try and "overshoot the mark" just in case something goes wrong. The worst thing that can happen is your don't design your trial correctly, with too few participants and you don't reach efficacy because you had too small of a sample. In effect, the more data you have, the more sure you are that your result is not a fluke. P values are obtained through statistical tests, and these tests check how different one data set is from another. You can use this kind of test to check lots of different things, it's used extensively in science, marketing and lots of other fields.
From what I can tell, overwhelming efficacy is usually assessed using a Haybittle-Peto boundary (https://en.wikipedia.org/wiki/Haybittle%E2%80%93Peto_boundary) which you can basically think of as a requirement that the trial is 10 times more significant than a strict pass mark, or p < 0.001 so that's "99.9% sure it works" in our laymen's terms. I'm not sure if the Haybittle-Peto boundary moves down if your trial significance benchmark changes from 0.05 to 0.01, perhaps there are others on this form that can advise on that?
Ease of achieving Haybittle-Peto efficacy at first readout (30% or 90 people):Assuming 35% mortality in the control group (16/45 deceased), the required mortality rate in the treated group for overwhelming efficacy would be < 9% (no more than 4 deceased of 45) for a p value of 0.0008. This is a pretty tough ask, in my opinion... though still technically possible. It would require both the control cohort to survive at the expected rate, and the treatment to be even more effective than in the early Mt Sinai trial. The effect of the control group on overwhelming efficacy cannot be understated, either. If even one more person survives than expected (mortality rate 33% or 15/45) then the treated group would need a mortality rate of no more than 6% to meet overwhelming efficacy requirements.
This is looking like it might be an outside shot unless we see mortality rates closer to 50% in the control group. Under this scenario the treated group mortality rate would only need to be no more than 17% for overwhelming efficacy.
Ease of achieving Haybittle-Peto efficacy at second readout (45% or 135 people):Assuming 35% mortality in the control group again (24/68 deceased), the required mortality rate in the treated group for overwhelming efficacy would be < 14% (no more than 9 deceased of 68) for a p value of 0.0009. Starts to get more interesting here. This result would be in the range of the Mt Sinai trial, and should be achievable if the results can be reproduced.
I would say this is a decent chance to achieve overwhelming efficacy if the control mortality is 35% and the treatment is as effective as we hope on these boards.
Ease of achieving Haybittle-Peto efficacy at third readout (60% or 180 people):Assuming 35% mortality in the control group once again (32/90 deceased), the required mortality rate in the treated group for overwhelming efficacy would be < 16% (no more than 14 deceased of 90) for a p value of 0.0007. I don't see this as a big change from the 45% readout, it might be that we just miss on the 45 and meet the objective here.
Ease of achieving full trial results efficacy (assuming a requirement of p < 0.001):Assuming 35% mortality in the control group one final time (52/150 deceased), the required mortality rate in the treated group for trial efficacy would be < 22% (no more than 33 deceased of 150) for a p value of 0.007. Much easier to achieve than the overwhelming efficacy requirements, and should be fairly safe provided the control group behave as expected, and the cells are as effective as other small trials have shown.
Hopefully this is helpful in painting some of the scenarios by which overwhelming efficacy is achieved. I personally wouldn't be disappointed if we missed the mark on the first readout, as it would require a huge result to pass the strict benchmarks used to halt the trial. There's plenty of criticism of Haybittle-Peto as a mechanism for judging overwhelming efficacy that it is too conservative (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6442021/) I don't know if I agree with that, it's good to be sure that the treatment passes the rigor of the scientific test, so that there aren't too many questions at the approval panel.
Summary table (assuming 35% control mortality):
| Stage | Control group mortality (number of people) | p < 0.05 mortality (number of people) | p < 0.01 mortality (number of people) | p < 0.001 mortality (number of people) |
|---|
| 1 | 30% | 16/45 | < 10 | < 7 | < 5 |
|---|
| 2 | 45% | 24/68 | < 16 | < 13 | < 10 |
|---|
| 3 | 60% | 32/90 | < 22 | < 19 | < 15 |
|---|
| 4 | 100% | 52/150 |
| < 34 | N/A |
|---|
Does anyone here know if they will publish the stats from the interim readouts? Or do we just get a "stop/continue" result?
Please DYOR on this, it's an important piece of the puzzle in being a MSB holder. As always, let me know if I've made a mistake in my stats here.
(20min delay)