Thanks for sharing the article!
I find the timing of this publication interesting, given that the results were based off studies conducted in Jun–Aug 2017, Feb–Sep 2018, and Apr–Sep 2019. I agree with the broader sentiment though... clearly the at-home sleep diagnosis market is heating up!
Whilst I haven't reviewed the contents of the journal publication in detail, nor do i profess to have a deep understanding of the science in this field, i have a few questions / comments regarding the high level comparison the authors' have made between the Firefly test set results vis-à-vis ResApp's results:
1.Is the standard of comparison between Firefly and ResApp's results fair? Reference to Firefly's performance is specific to cases where AHI≥15 events/hour (e.g. moderate or severe OSA cases). In addition, the Firefly test set results for AHI≥15 events/hour (sensitivity of 88.3% and specificity of 80.0%) do not distinguish between whether the patient has moderate or severe OSA, but rather just whether they have either. Furthermore, how did Firefly's tech perform in regards to the identification of mild sleep aponea?
ResApp's sensitivity and specificity and AUC results were specific to the three separate categories mild, moderate and severe OSA. My understanding is that a distinction between whether the person has mild, moderate or severe sleep aponea is extremely important for screening purposes, as the recommended next steps will be based off the underlying risk category the patient fits within.
To this point, the author's note the following in regards to the Firefly results: "
The system also provides an estimate for the AHI score, which exhibits good performance as depicted in Table 5 and Figure 12, with a correlation of 0.81 on the testing set. Specifically, 61.7% (74/120) of the AHI results from the Firefly technology match their correspondent PSG result in the class of OSA severity (none, mild, moderate or severe). Encouragingly, where differences were detected, the errors are limited to the adjacent class of OSA severity in 95% of the cases, with only 6 recordings out of 120 showing difference of OSA severity class prediction larger than one level. The system has good specificity with only two normal subjects being classified as moderate."
As such, it seems that the Firefly test set results for AHI≥15 events/hour (sensitivity of 88.3% and specificity of 80.0%), might not be based off a comparable standard to the quoted ResApp's results (sensitivity of 86% and specificity of 83%). I personally would like to understand more regarding the impact of this distinction and what the overall result would be if the two were compared on a more like-for-like basis.
2.How does Firefly perform in an at-home testing?The reference standard used in the Firefly study is the gold standard full attended PSG system. This was the standard that ResApp originally used to evaluate the performance of our tech in a study completed in 2018. However, recognising that ResApp is designed to be an At-Home-Testing screening tool and to support the "final step" towards our regulatory submissions, ResApp then completed a further study to evaluate the performance of our tech against a full
unattended (at home) PSG (results published in 2019).
As such, I would assume that Firefly's technology should also be subject to similar at-home testing settings prior to regulatory submissions being filed for its use as an at-home screening tool.
The author's in fact note the following regarding the limitations of their laboratory studies: "
It is noted that the constrained nature of the laboratory setting in ASR might not be fully representative of the variability of setup conditions that can be encountered in a home environment. However, some variability in room type was provided, with different sizes and configuration, as well as air conditioning being used on some nights (a potential interferer with other acoustic systems). Furthermore, ResMed has prior experience in deploying contactless sleep measurement in the home environment in unsupervised conditions, and managing the associated variability, such as with the S+ by ResMed RF and SleepScore Max bedside devices (46)."
Regardless of the weight you place on the last sentence regarding ResMed's prior experience in deploying contactless sleep measurement in the home environment in unsupervised conditions, I think this is a relevant limitation to the Firefly test results that should be called out.
3. Robustness of the Firefly resultsI'd like to understand the impact of the data size and the method of evaluation has on the robustness of the Firefly test set results and how this compares to ResApp's data.
According to the journal, the Firefly results are based off training and test set results, of which only data from 120 patients was analysed as part of the Firefly testing set. Whereas in regards to ResApp, quoting Dr Philip Currie in one of ResApp's announcements (dated 30 Sep 2019) "...this very large study, which recruited over 2,000 patients, conducted both in the sleep laboratory and at home has developed and validated a simple and accurate screening tool using only a smartphone in the bedroom. More specifically according to ResApp's annoucements, across the two published ResApp sleep study results (2018 and 2019 results), data from 582 patients in the first study and 238 patients in the second study were analysable. As such, the Firefly results are based off a much smaller sample size than the ResApp studies, which I assume would impact the comparative statistical power / significance of the Firefly results.
If anyone has expertise / knowledge to comment on any of the above topics in more detail, your feedback would be much appreciated. Regardless of my comments above, the Firefly technology undoubtedly seems to be very powerful. I'm glad ResApp is currently ahead in terms of commercialisation, but it definitely goes to show that we need to get SleepCheck approved and circulating in the USA ASAP.
Cheers
