setting the record straight, page-18

Hi Southoz, the way I wrote the paragraph beginning with "Regarding failure to publish" has certainly lead to my confusing the point, and has caused others to be confused by what I meant. For the record, I was attempting to discuss peer review in regard to scientific merit, not that the FDA approval process requires such a thing as evidence of efficacy. I hope my lack of clarity on this point can be forgiven! I will note on this point that the FDA tend not to approve treatments off the results of data mined statistics but more on the reasons for this below.

I must confess that I noticed this potential for misinterpretation on that point last week during a discussion I had on this topic via email, but I've unfortunately been busy and not had the time to come clear up any confusion. Thank you Southoz for helping clear this up in my absence!

For all those wishing to get a more in-depth analysis of the concerning issues surrounding a lack of publication and peer review for FDA approved treatments, please see the following article: http://jama.ama-assn.org/content/305/17/1786.abstract.

Now on the topic of data mining, I have to disagree with Southoz. While I do agree that there is certainly value in obtaining as much data as possible from a clinical trial, a post-hoc data mining approach severely diminishes the statistical power of any additional analyses performed, and you are likely to throw out a statistically significant result in 1 in 20 analyses. Realistically, you have to repeat the analysis in another study with this correlation defined as a primary endpoint. As for how this affects MSB, more on that later!

My reason for discussing this point in the initial post was to explain the following, which I fear I still did not express adequately: Mining for data is fine, but it can only be used to design future studies, and cannot reliably tell us anything about studies that have already been performed.

Now, if MSB's phase III studies are being designed with the endpoints above in mind, then there is no problem. So, you should all be very happy to know that the Phase II trial design does stipulate the following (from the clinical trials register): "The secondary objectives are to explore functional efficacy for subsequent study design". This is very good news, and shows that MSB are approaching the design of their trials in a manner which has scientific merit.

Now, everyone bear in mind that regardless of this you cannot rely on the information you have at the moment as evidence of efficacy, and discovering encouraging efficacy results obtained from mining all available data points relating to response does not mean that the Phase II trial was successful.

What does make MSB's trial successful is that it achieved its primary endpoint of showing a lack of adverse side effects! This is very good! The phase III trial results which will actually have to show efficacy, and you should take the current efficacy stats with a grain of salt - particularly given the size of the trial, and the small number of events in some of the analyses.

Knuhfoti is certainly correct when he comments that at the end of the day the FDA is who chooses which drugs are approved for sale. However it is important to recognise that failure to publish results does not bode well for the efficacious merit of a treatment. The paper Southoz linked is just the tip of the iceberg as far as bias and reporting of results is concerned, and this TED talk by Ben Goldacre may elucidate the extent of the issue. I could go on for hours about why I am rather reluctant to place my hard earned in support behind data which has not been published, but hopefully my reasons are clear!

Under normal circumstances I'd be interested in analysing the patient cohorts in the study in detail (what's a man got to do for a kaplan-meier plot around here?!), but this isn't possible with MSB's data at the moment.

A prominent example from the ANN is the following: "only 1/45 patients who received Revascor™ died of cardiac causes compared with 3/15 controls" - four events is not very much data to go on, particularly with mined data, and when coupled with a lack of any detailed survival data or other information relating to the individual patients and the patient cohorts, this is not a very convincing statistic.

Also, "improved functional capacity (gain of 52.6 meters over 6 minutes’ walk... p=0.06)" is not a statistically significant result, but could perhaps be due to the small sample size.

Bear in mind the Phase II study was not double-blind, but only blinded to patients.

Someone asked why I'm here posting when I don't hold stock, and asking about my inspiration for writing these responses, well I hope people would notice that while my position is "No Stock Held", my sentiment is not "Sell". My sentiment is "None". I'm on the sidelines at the moment, waiting to be convinced.