Post-market evaluation of serology-based point of care tests
29 September 2020In response to the COVID-19 pandemic, the Therapeutic Goods Administration (TGA) has expedited the approval (with conditions) of COVID-19 tests including tests intended for laboratory use and those intended for use by specified health professionals at the point of care (POC). Approval was based on the studies manufacturers were able to provide at the time of application.
Accurate identification of a COVID-19 infection based on serology results, particularly those obtained from tests used at the POC, requires an understanding of the antibody response profile which is currently not well defined. It is known that these tests can fail to detect COVID-19 if testing is performed in the acute phase of the infection prior to the development of detectable antibodies. If this 'negative' result is incorrectly interpreted the person may be told they haven't been infected when they may have been infected and may be infectious. Results need to be interpreted, in conjunction with clinical presentation, by a suitably qualified health care professional who can provide a patient with appropriate advice and treatment if required.
As part of a Post Market Review the TGA is now reviewing all approved serology-based COVID-19 POC tests to verify their ability to detect antibodies to SARS-COV-2 (the virus that causes COVID-19), taking into consideration the timeframes for an individual to develop detectable levels of antibody. The Peter Doherty Institute for Infection and Immunity (the Doherty Institute), has been engaged by the Department of Health to assist with the post-market verification process of these tests, to inform their best use. However reports by other Australian laboratories or comparable international regulators may also be taken into consideration.
To date, the Doherty Institute(link is external) has completed its validation of thirteen different serology-based POC tests approved by the TGA. Samples of the thirteen POC tests were provided from the following sponsors (note some devices have multiple sponsors):
- MD Solutions Australasia Pty Ltd, sponsor of the Onsite IgM/IgG Rapid Test manufactured by CTK Biotech Inc (USA);
- Endo X, sponsor of the VivaDiag COVID-19 IgM/IgG Rapid Test manufactured by VivaChek Biotech (Hangzhou) Co Ltd (China);
- AM Diagnostics, sponsor of the 2019-n-CoV IgG/IgM Rapid Test Cassette manufactured by Hangzhou Alltest Biotech Co Ltd (China);
- Cellmid Limited and Tayler Dental Consulting Pty Ltd, sponsors of the SARS-CoV-2 Antibody Test (Lateral Flow Method) manufactured by Guangzhou Wondfo Biotech Co Ltd (China);
- Envon Pty Ltd, sponsor of the SARS-CoV-2 IgM/IgG Antibody Rapid Test manufactured by Qingdao Hightop Biotech Co Ltd (China);
- Biovent Consulting Pty Ltd, sponsor of the GenBody COVID-19 IgM/IgG manufactured by GenBody Inc (Korea);
- Aurora Future Education Group, sponsor of the Innovita 2019-nCoV Ab Test (Colloidal Gold) manufactured by Innovita (Tangshan) Biological Technology Co Ltd (China); and
- Life Clinic Australia Pty Ltd, sponsor of the Wantai SARS-CoV-2 Ab Rapid Test kit manufactured by Beijing Wantai Biological pharmacy Enterprise Co Ltd (China);
- Haemokinesis Pty Ltd, sponsor of the PCL COVID19 IgG/IgM Rapid Gold, manufactured by PCL Inc (Korea);
- Medicision Pty Ltd, sponsor of the Cellex qSARS-CoV-2 IgG/IgM Rapid Test kit, manufactured by Cellex Inc (USA);
- Expia Pty Ltd and Onsite Diagnostics Pty Ltd, sponsors of the COVID 19 IgG/IgM Rapid Test Cassette, manufactured by Zhejiang Orient Gene Biotech Co Ltd (China);
- Complementary Medicines Group Pty Ltd, sponsor of the LYHER Novel Coronavirus (2019-nCoV) IgM/IgG Antibody Combo Test Kit, manufactured by Hangzhou Laihe Biotech Co Ltd (China); and
- Marcel Equity Pty Limited/Avania PL, sponsor of the Diagnostic Kit for IgM/IgG Antibody to Coronavirus (SARS-CoV-2), manufactured by Zhuhai Livzon Diagnostics Inc (China).
Doherty Institute's validation
The validation performed by the Doherty Institute for these thirteen devices suggests that manufacturers have claimed a better sensitivity compared to that observed in the Doherty Institute studies (when compared to a molecular-based method). The Doherty Institute studies did demonstrate that the sensitivity of the eight tests improved with increasing duration (i.e. longer time) between sample collection and symptom onset up to approximately 20-30 days post-symptom onset.
Onsite IgM/IgG Rapid Test (CTK Biotech Inc, USA)
The overall total antibody (i.e. either IgM or IgG) sensitivity and specificity claimed by the manufacturer in the instructions for use (IFU) provided with the device is 96.86% and 99.39% respectively. The information provided in the IFU was silent on the relative sensitivity of the test at different time points post symptom onset.
The overall total antibody sensitivity of the device when tested by the Doherty Institute was found to be 56.9% while the total antibody specificity was found to be 95.6%. For samples collected 14 days or longer post onset of symptoms the overall sensitivity improved to 84.6%.
VivaDiag COVID-19 IgM/IgG Rapid Test (VivaChek Biotech (Hangzhou) Co Ltd, China)
The overall total antibody sensitivity claimed by the manufacturer in the IFU provided with the device is 81.25% between days 4 to 10 and 97.1% between days 11 to 24. Overall claimed total antibody specificity was 100%.
The overall total antibody sensitivity of the device when tested by the Doherty Institute was found to be 51.8% while the total antibody specificity was found to be 97.8%. For samples collected 14 days post onset of symptoms the overall sensitivity improved to 78.5%.
2019-n-CoV IgG/IgM Rapid Test Cassette (Hangzhou Alltest Biotech Co Ltd, China)
The sensitivity and specificity claimed by the manufacturer in the IFU provided with the device for IgM is 85.0% and 96.0% respectively; and for IgG, 100% and 98.0% respectively.
The overall total antibody sensitivity of the device when tested by the Doherty Institute was found to be 60.6% while the total antibody specificity was found to be 96.7%. For samples collected 14 days post onset of symptoms the overall sensitivity improved to 90.8%.
SARS-CoV-2 Antibody Test (Lateral Flow Method) (Guangzhou Wondfo Biotech Co Ltd, China)
The overall total antibody sensitivity and specificity claimed by the manufacturer in the IFU provided with the device is 86.43% and 99.57% respectively (note this device does not distinguish between IgM and IgG results).
The overall total antibody sensitivity of the device when tested by the Doherty Institute was found to be 68.6% while the total antibody specificity was found to be 97.8%. For samples collected 14 days post onset of symptoms the overall sensitivity improved to 93.8%.
SARS-CoV-2 IgM/IgG Antibody Rapid Test (Qingdao Hightop Biotech Co Ltd, China)
The overall total antibody sensitivity and specificity claimed by the manufacturer in the IFU provided with the device is 94.15% and 93.91% respectively.
The overall total antibody sensitivity of the device when tested by the Doherty Institute was found to be 61.0% while the total antibody specificity was found to be 100%. For samples collected 14 days post onset of symptoms the overall sensitivity improved to 93.8%.
GenBody COVID-19 IgM/IgG (GenBody Inc, Korea)
The overall total antibody sensitivity (i.e. either IgM or IgG) claimed by the manufacturer in the IFU is 89.3% and 88.0% for samples collected 7 days from symptom onset. Overall claimed total antibody specificity was 95.9%.
The overall total antibody sensitivity of the device when tested by the Doherty Institute was found to be 38.5% while the total antibody specificity was found to be 100%. For samples collected 14 days post onset of symptoms the overall sensitivity improved to 58.5%. Based on the Doherty Institute’s findings, there was a significant difference in performance of this device in comparison to similar devices evaluated as part of the post-market review and the claims made in the IFU.
Innovita 2019-nCoV Ab Test (Innovita (Tangshan) Biological Technology Co Ltd, China)
The overall total antibody sensitivity and specificity claimed by the manufacturer in the IFU is 85.0% and 97.4% respectively (compared to RT-PCR).
The overall total antibody sensitivity of the device when tested by the Doherty Institute was found to be 39.4% while the total antibody specificity was found to be 100%. For samples collected 14 days post onset of symptoms the overall sensitivity improved to 56.9%. Based on the Doherty Institute’s findings, there was a significant difference in performance of this device in comparison to similar devices evaluated as part of the post-market review and the claims made in the IFU.
Wantai SARS-CoV-2 Ab Rapid Test kit (Beijing Wantai Biologicalpharmacy Enterprise Co Ltd, China)
The overall total antibody sensitivity and specificity claimed by the manufacturer in the IFU is 94.70% and 98.89% respectively.
The overall total antibody sensitivity of the device when tested by the Doherty Institute was found to be 82.0% while the total antibody specificity was found to be 98.0%. For samples collected 14 days post onset of symptoms the overall sensitivity improved to 84.2%.
PCL COVID19 IgG/IgM Rapid Gold (PCL Inc, Korea)
The overall total antibody sensitivity and specificity claimed by the manufacturer in the IFU is 81.35% and 99.37% respectively.
The overall total antibody sensitivity of the device when tested by the Doherty Institute was found to be 48.0% while the total antibody specificity was found to be 100.0% For samples collected 14 days post onset of symptoms the overall sensitivity was 50.0%.
Cellex qSARS-CoV-2 IgG/IgM Rapid Test kit (Cellex Inc, USA)
The overall total antibody sensitivity and specificity (i.e. either IgM or IgG) claimed by the manufacturer in the IFU is 93.8% and 95.6% respectively. The sensitivity and specificity claimed by the manufacturer in the IFU for IgM is 82.0% and 98.4%, respectively; and for IgG is 90.6% and 97.6%, respectively.
The overall total antibody sensitivity of the device when tested by the Doherty Institute was found to be 76.0% while the total antibody specificity was found to be 97.0% For samples collected 14 days post onset of symptoms the overall sensitivity improved to 86.8%.
COVID 19 IgG/IgM Rapid Test Cassette (Zhejiang Orient Gene Biotech Co Ltd, China)
The sensitivity and specificity claimed by the manufacturer in the IFU for IgM is 87.9% and 100.0% respectively; and for IgG, 97.2% and 100.0% respectively.
The overall total antibody sensitivity of the device when tested by the Doherty Institute was found to be 90.0% while the total antibody specificity was found to be 99.0% For samples collected 14 days post onset of symptoms the overall sensitivity improved to 97.4%.
LYHER Novel Coronavirus (2019-nCoV) IgM/IgG Antibody Combo Test Kit (Hangzhou Laihe Biotech Co Ltd, China)
The overall total antibody sensitivity and specificity claimed by the manufacturer in the IFU is 96.98% and 99.29% respectively.
The overall total antibody sensitivity of the device when tested by the Doherty Institute was found to be 80.0% while the total antibody specificity was found to be 97.0% For samples collected 14 days post onset of symptoms the overall sensitivity improved to 86.8%.
Diagnostic Kit for IgM/IgG Antibody to Coronavirus (SARS-CoV-2) (Zhuhai Livzon Diagnostics Inc, China)
The overall total antibody sensitivity and specificity (i.e. either IgM or IgG) claimed by the manufacturer in the IFU is 90.6% and 99.2% respectively. The sensitivity and specificity claimed by the manufacturer in the IFU for IgM is 79.0% and 99.7%, respectively; and for IgG is 84.3% and 99.4%, respectively.
The overall total antibody sensitivity of the device when tested by the Doherty Institute was found to be 58.0% while the total antibody specificity was found to be 99.0% For samples collected 14 days post onset of symptoms the overall sensitivity was 57.9%.
It should be noted that none of the manufacturers claim that these tests should be used as a sole basis for diagnosis of COVID-19 and advise that results need to be interpreted along with other clinical findings and consideration should be given to further testing.
Some of the variation observed between the different manufacturers' claims and the Doherty Institute's findings can be attributed to sample size and sampling bias and it is apparent that different patient populations and samples taken at different times post onset of symptoms can significantly influence the performance of the tests. The Doherty Institute also noted that a direct comparison with the manufacturers' claims was difficult as the performance information provided in the IFUs was limited. However four of the devices, the GenBody COVID-19 IgM/IgG, the Innovita 2019-nCoV Ab Test, PCL COVID19 IgG/IgM Rapid Gold and the Diagnostic Kit for IgM/IgG Antibody to Coronavirus (SARS-CoV-2), produced results that differed significantly from the manufacturer's claims and performed poorly in comparison to similar devices evaluated by the Doherty Institute.
The Doherty Institute's studies, along with other evaluations performed by Australian and international laboratories confirm that the sensitivity of these tests in the early stages of infection is poor and that they shouldn't be used for the diagnosis of acute COVID-19 infection. This conclusion is consistent with current advice from the Public Health Laboratory Network, the Royal College of Pathologists of Australasia and the TGA that these tests must be used with caution due to the potential for these tests to fail to detect COVID-19 during the acute phase of the illness, prior to the development of antibodies.
Based on the Doherty Institute's findings it is recommended that serology-based COVID-19 POC tests should not be used until at least two weeks post onset of symptoms.
As yet not enough is known about the adequacy of the COVID-19 immune response or duration of immunity. Whether there is a role for these tests in determining immunity for return to work purposes or for population-level surveillance remains to be seen.
Post-market regulatory action
The TGA has requested the sponsors of these devices to provide further information on the clinical studies undertaken by the manufacturer to support the performance of these devices. This includes a re-evaluation of the clinical performance studies including stratification of results according to different periods post onset of symptoms and will provide a better basis for comparing the manufacturer's claims against the Doherty Institute's findings. This will inform the need for any subsequent regulatory action which may include changes to the IFUs to include more detail about the test sensitivity for samples collected at different time points post onset of symptoms; and additional warnings to inform the user of the limitations of test. Other regulatory action includes suspension or cancellation of non-compliant or poor performing tests from the ARTG. Further information can be found via the post market review of COVID-19 point-of-care tests page.
