Goodness me. You have not answered my questions at all.
For anyone invested in IMU, the following lines of information need to be read very carefully. Many here have tried to undermine what I say by insulting me or saying I do not know more than the IDMC, however, what I am about to share with you is data that I have extracted from credentialed researchers around the world, and the Food and Drug Administration (FDA) and the European Medicines Agency (EMA) guidance papers. I will provide references for everything I say and encourage anyone to debunk my work using references and interpretations of their own.
Null hypothesis? That’s rubbish! Unless you are simply referring to safety data, where there was no added toxicity over SOC which was chemo in this instance. So that is actually a positive statement.Incorrect. A null hypothesis is not rubbish. Rothman et al. states "Null hypothesis testing is a foundational stone of statistical analysis" (1). Hypothesis testing uses statistical analyses to determine if there is an effect within the data (2). In research, there are two hypotheses:
H0 (the null hypothesis): there is no difference between the study arms
H1 (the alternative hypothesis): there is a difference between the study arms
The null and alternative hypotheses are mutually exclusive, which means that only one can be true. To determine which is true requires the use of statistical analyses. There are two common statistical analyses metrics used to accept or reject the null hypothesis include the p value and confidence intervals.
The p value is quite simple to interpret, where if the p value is below the level of significance set at the beginning of the study (either 0.01, 0.05, or 0.10), then you may reject the null hypothesis and therefore accept the alternative hypothesis (3). In other words, a p value lower than 0.05, for example, indicates that a relationship is significant and there is indeed a difference between study arms. However, if the p value is above the level of confidence (or not significant), then you accept the null hypothesis (that there is no difference between the study arms). If anyone on this forum would like to dispute the importance or relevance of the p value (
particularly the significance level of 0.05), then you can first debunk why it has been used for almost a century in research and why the 0.05 significance level has been a cornerstone in FDA decision making for 50-years (3; below).
The confidence interval can be more complex, so I have shared an screenshot of a placebo-controlled superiority studies figure from an FDA guidance report (4). The confidence interval can be anywhere from 50 to 99%, however the 95% confidence interval has long been considered the FDA's and EMA's standard for reporting of clinical data (5,6). Research organisations like to also use confidence intervals when reporting data, as there are some limitations with using the p value on its own (3). A p value will tell you whether the relationship is significant or not, whereas a confidence interval will show a range of data where a person could be x% confident the true figure would lie. This is advantageous because it provides a deeper insight into the data. Looking towards the figure below you will see that there are three lines on the graph. Basically, if that line crosses '1', it indicates that the null hypothesis is within the bounds of the 95% confidence interval, therefore you accept the null hypothesis. If the line does not cross '1', you reject the null hypothesis. If you are following, then you will see that test 1 indicates the drug is effective (reject null hypothesis and accept alternative), whereas test 2 and 3 demonstrate the drug is not different to the control (accept the null hypothesis). Basically, if the confidence interval includes 1, then you can accept the null hypothesis - there is no difference between study arms.
Now, let's turn our attention towards the recent HER-Vaxx PFS data that was released recently and actually answer my question with data. Based on the announcement, we can be sure that the stastitical analyses metrics that IMU decided to use include a
1-sided p value of 0.10 (7; first screenshot). They do not quote a confidence interval in their final PFS data released to market. Fortunately, the recent abstract publication of the phase 2b HER-Vaxx interim data indicates an 80% confidence interval was used for the PFS data (8; second screenshot).
Now, the questions I have asked you,
@Owl vs Fox, are whether you accept of reject the null hypothesis, and what statistical analyses metric you used to determine this. The correct answer is as follows:
For progression free survival (PFS), we
accept the null hypothesis (that there is no difference between the study arms), as the p value of 0.266 is not significant (>0.10) (
statistical analysis metric). Also, given that the interim PFS data reflected a HR of 0.532 with an 80% confidence interval of 0.267 and 1.060 (
intervals which cross '1'; accept null hypothesis), we can be more confident of accepting the null hypothesis (there is no difference between the study arms). This is because the PFS HR has decreased from 0.532 to 0.719 over the space of 5-months and the 80% confidence interval would have further included '1'. Also, I believe that this increase in PFS HR is due to patients developing therapy acquired resistance to HER-Vaxx, which I had discussed as something I noticed in the HER-Vaxx phase 1 trial about 4 months ago (
here). In short, HER-Vaxx has failed to show a significant benefit over standard of care chemotherapies for progression free survival in this patient population.
There are additional points that need to be considered when evaluating this data.
1. The use of an 80% confidence interval instead of the industry standard of 95%. An 80% confidence interval tightens the intervals and increases the likelihood that there will be inconsistencies in the data, whereas an 95% confidence interval is of high academic standard due to the difficulties in establishing tight intervals (9; screenshots below). For comparison, the Herceptin TOGA study that IMU have routinely quoted used a 95% confidence interval in their reporting, where they found a statistically significant improvement over standard of care chemotherapies (the 95% CI does not include 1, therefore the null hypothesis is rejected in favor of the alternative) (10; below).
Something that may be of interest to those who are prepared to look at what I am saying objectively. You may remember that the final HER-Vaxx PFS data did not state the confidence interval. If you are curious about your investment, you may be wondering why. I recommend you send Leslie Chong an email asking the following questions, as I did:
- I'm wondering if you could please provide the 95% confidence intervals for the PFS data you released on the 1st of September.
- Could you also please explain why the 95% confidence intervals were left out of the announcement to market.
In short, I was told by Leslie Chong that she had been adviced to not talk directly to shareholders. Maybe someone else will have more luck than I in these matters. As I have discussed, confidence intervals are essential to making complete sense of the data.
2. The use of a one-sided t test to determine the effectiveness of HER-Vaxx compared to SoC drugs. There are either one-sided or two-sided t tests, and both have their place in research. The issues with one-sided t-tests in drug development is that it only tests the significance in one direction (11). In the case of the HER-Vaxx trial, the researchers are only interested in testing whether HER-Vaxx is better than the SOC and not worse. The screenshot below explains why this is a problem (11). For comparison, the Herceptin TOGA study have used two-sided t tests to determine all of their p values, highlighting their advanced stastistical analyses compared to the HER-Vaxx trial (11; below).
3. The use of a significance threshold of 0.10. If you have been following, then you will realise that a significance level of 0.10 is well above the FDA and EMA recommendations of 0.05 (3,5,6), highlighting the lowered stastistical standards of the HER-Vaxx trial. For comparison, the Herceptin TOGA study used a significance level of 0.05, where the p value for the PFS data was 0.0002 (well below the 0.05 cut off, indicating highly significant data) (11; below).
Based on this data it is safe to assume that in this patient population, HER-Vaxx is no different from standard of care chemotherapy. HER-VAXX recruitment was stopped early on IDMC recommendation for safety & efficacy in a Ph2. That is not done lightly, nor a common occurrence. So I’m accepting of the published P value.Yes, the IDMC recommended that the HER-Vaxx trial be discontinued early. The IDMC rely on the statistical analyses underpinning the trial to make their conclusions about stopping a clinical trial early. As I have discussed above, the statistical analyses used in the HER-Vaxx trial are well below the FDA and EMA standards and as such, are non-comparable to the Herceptin TOGA study data. Thus, efficacy being the primary driver of early cessation is highly questionable. Of course the drug is not going to add toxicity if it failing to show a significant benefit over the standard of care treatments.
If you accept the published HER-Vaxx PFS HR p value (p = 0.226), you therefore also accept the null hypothesis - that there is no difference between the study arms. I'm glad that we agree on this.
Very simply, when reviewing published data on clinical trials, you really need to look at each trial individually & the stage of the trial & number of participants & the primary & secondary endpoints, of which there can be many parameters in Immuno-oncology.Absolutely incorrect. There are fundamentals that are industry standard accross multiple different areas of research to ensure quality of reporting. Here are some examples of FDA and EMA guidance documents highlighting 95% confidence intervals and significance set to 0.05 in clinical research, as well as a good many other things.
(FDA) Non-Inferiority Clinical Trials to Establish Effectiveness (4)
(FDA) Providing Clinical Evidence of Effectiveness for Human Drug and Biological Products (12)
(EMA) GUIDELINE ON THE CHOICE OF THE NON-INFERIORITY MARGIN (5)
So for example a trial on 15 participants on an Orphan status trial that has shown efficacy, you cannot expect to have statistical significance, nor are such trials powered for that. You need to look at results.HER-Vaxx are not functioning within an orphan status space. IMU are competing against a drug (Herceptin) that has been established in the market for decades. HER-Vaxx needs to be better than Herceptin to make it into the multi-billion dollar US markets, or it will be a failure. At this point in time, HER-Vaxx cannot demonstrate significant benefit over standard of care chemotherapy, let alone Herceptin. This also highlights why strong statisitical analysis methods are required - to unequivocally prove that HER-Vaxx is better than Herceptin.
I have looked at the results in depth and shared my research publicly. I have extensively compared the findings of HER-Vaxx phase 2b trial to industry standards and the Herceptin TOGA study, and found major holes that no one is addressing. Maybe it is you that needs to look at the results.
---
HER-Vaxx failed to show a significant benefit over standard of care chemotherapy for PFS in this patient population.
I encourage anyone on this forum to take the time to prove where that has not happened.
The rest of your comments are not why I am here. If you have a problem with a contrarian view, that's not my problem. I have 0 'ill wishes' towards IMU holders and instead want to discuss the fundamentals of a stock that thousands of people have invested their hard earned money into.
1
https://www.tandfonline.com/doi/full/10.1080/19466315.2021.18861642
https://www.investopedia.com/terms/h/hypothesistesting.asp3
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6169785/pdf/nihms-987338.pdf4
https://www.fda.gov/media/78504/download5
https://www.ema.europa.eu/en/documents/scientific-guideline/guideline-choice-non-inferiority-margin_en.pdf6
https://chicagounbound.uchicago.edu/cgi/viewcontent.cgi?referer=&httpsredir=1&article=1148&context=public_law_and_legal_theory7
https://newswire.iguana2.com/af5f4d73c1a54a33/imu.asx/3A574809/IMU_Imugene_HER-Vaxx_Phase_2_PFS_Data_and_Flags_Three_New_Trials8
https://ascopubs.org/doi/abs/10.1200/JCO.2021.39.15_suppl.e160659
https://measuringu.com/confidence-levels/10 https://sci-hub.se/https://www.thelancet.com/journals/lancet/article/PIIS014067361061121X/fulltext
11
https://stats.idre.ucla.edu/other/mult-pkg/faq/general/faq-what-are-the-differences-between-one-tailed-and-two-tailed-tests/12
https://www.fda.gov/files/drugs/published/Providing-Clinical-Evidence-of-Effectiveness-for-Human-Drug-and-Biological-Products..pdf
(20min delay)
