Short version for those who want to get to the punch line quickly.
A “purported” newsletter from Project Transformer was posted on HC on Friday. The newsletter contained a lay summary of the results from the Tali detect study intended for participants who had taken part in the study.
I have seen many lay summaries of study results and written a few myself. I can spot a fake a mile away. Personally I am 100% convinced the newsletter was legitimate.
The results presented in the newsletter indicated that many Tali detect sub-tasks showed poor levels of reliability and validity. I concluded that this meant that Tali detect total scores would have poor reliability and validity. If Tali detect is not reliable or valid arguably it should not be being used to entice parents into signing up for the Tali train program. Tali detect is what feeds parents into signing up to the Tali train program; hence the commercial sensitivity.
There ends the short version.
The long version follows for those interested in why I think NHL has gotten into this mess.
In December 6 2017 Novita announced that it had been awarded $1.2m (from the Commonwealth government) to design, develop and commercialise Tali detect as part of a CRC project. Total funding for the project (with additional funding from project partners) was $3.9m.
In the ASX announcement Tali detect was expected to:
“… enable cost-effective screening for ADHD in pre-school children, helping to identify those who could benefit from Novita’s TALI Train software”
Break down these key elements. Cost effective because the test is administered by parents it avoids expensive clinician testing. Screening for ADHD in children to identify those who do or do not have ADHD. Benefit from Tali train - Tali train is a treatment program for children with ADHD.
This description provided to the ASX however is inconsistent with the official description of the project. According to the official version the project was to assess inattention difficulties. ADHD is not mentioned at all.
Right from day one the researchers and industry partners are not quite on the same page. And I think this sows the seeds for Fridays cock-up.
Fast forward a year from the partnership grant and the researchers explain what they think the Tali detect study is trying to do:
“The researchers hope TALI Detect will help identify subclinical levels of attention difficulties in young children that may not otherwise be picked up.” “We don’t look at it in terms of ‘disorder’. We think of it as having inattention vulnerabilities, meaning they may struggle with certain aspects of learning,” says Research Fellow James Kean”.
Break down these elements. “Subclinical levels of attention” means attention problems that would not lead to a diagnosis of ADHD. The researchers don’t look at it in terms of “disorders”. Inattention vulnerabilities” means the child may not even be showing any attention problems at all but they are “vulnerable” and so may benefit for Tali train “attention strengthening” exercises.
This is a long way from what the company thought was going on in 2017. And a long way from what posters on here seem to think … that Tali train is treatment program for children with ADHD.
If Tali train was a treatment program for children with ADHD the Tali detect study design is simple. Tali detect should identify children with / without ADHD with good levels of sensitivity / specificity. Children who test positive in Tali detect then go on to the Tali train treatment program for ADHD.
But this is not what the researchers have done at all. If they haven’t done a diagnostic accuracy study what then have they done? Standard psychometric testing.
First is Tali detect a reliable test? By reliable the researchers mean that Tali detect results two weeks apart should be correlated. In other words, Tali detect should be measuring a stable “trait” of attention vulnerabilities” not simply whether a child is having a good or bad attention day (a state).
Second criterion validity. The results from Tali detect should correlate with a gold standard test of attention difficulties. In this case the TEA-ch. This clinician administered test was the primary outcome measure for the Tali train efficacy studies; and is widely recognised as a “gold standard. The researchers hope that the Tali detect subtasks will correlate well with the equivalent TEA-ch subtasks.
Tali detect comprises a number of sub-tasks. Across most sub-tasks of Tali detect the results presented in the newsletter showed poor test retest reliability and poor criterion validity. The researchers shy away from presenting total scale reliability and validity scores. But classical test theory is unlikely to save the day here; the extent of the poor results at the sub-task level are suggestive of poor reliability and validity at the total score level.
The researchers seem to think they can tweak the sub-tasks to improve reliability and validity. I doubt this. The good levels of reliability / validity from the gold standard TEA-ch are produced from highly trained and skilled clinicians under optimum (and this can be difficult to achieve even for the experts) testing conditions. It is unrealistic to think that your average parent can match this. That is where the problem will lie here.
And the difficulty is exacerbated by the challenge of testing for “attention vulnerabilities” as opposed simply testing for a binary yes / no screening type for ADHD where you can set a high sensitivity but low specificity cut-point.
What I think the researchers have proven is that if you want a reliable and valid assessment of your child’s attention you should pay a few hundred dollars for a properly trained health professional to administer a gold standard test.
If it were otherwise there would be an awful lot of grumpy psychologists around who have paid a couple of thousand dollars for the TEA-ch package (the co-developer of this is also a Tali detect co-investigator) when the parent can DIY on the internet for free.
Parents should carefully read the Tali detect disclaimer. To me this is saying you are very much going to get what you pay for from a fee internet test.
Posters here will appreciate I have been negative about NHL right from the start and will take this all with a strong pinch of salt. That’s fine … think of this as the case for the prosecution. Look forward to seeing the case for the defence from the company.
NHL Price at posting:
5.4¢ Sentiment: Sell Disclosure: Not Held
(20min delay)