Describing the Reportable Range is Important for Reliable Treatment Decisions

Because interpretation of next-generation sequencing (NGS) data remains challenging, optimization of the NGS process is needed to obtain correct sequencing results. Therefore, extensive validation and continuous monitoring of the quality is essential. NGS performance was compared with traditional detection methods and technical quality of nine NGS technologies was assessed. First, nine formalin-fixed, paraffin-embedded patient samples were analyzed by 114 laboratories by using different detection methods. No significant differences in performance were observed between analyses with NGS and traditional techniques. Second, two DNA control samples were analyzed for a selected number of variants by 26 participants with the use of nine different NGS technologies. Quality control metrics were analyzed from raw data files and a survey about routine procedures. Results showed large differences in coverages, but observed variant allele frequencies in raw data files were in line with predefined variant allele frequencies. Many false negative results were found because of low-quality regions, which were not reported as such. It is recommended to disclose the reportable range, the fraction of targeted genomic regions for which calls of acceptable quality can be generated, to avoid any errors in therapy decisions. NGS can be a reliable technique, only if essential quality control during analysis is applied and reported. Because interpretation of next-generation sequencing (NGS) data remains challenging, optimization of the NGS process is needed to obtain correct sequencing results. Therefore, extensive validation and continuous monitoring of the quality is essential. NGS performance was compared with traditional detection methods and technical quality of nine NGS technologies was assessed. First, nine formalin-fixed, paraffin-embedded patient samples were analyzed by 114 laboratories by using different detection methods. No significant differences in performance were observed between analyses with NGS and traditional techniques. Second, two DNA control samples were analyzed for a selected number of variants by 26 participants with the use of nine different NGS technologies. Quality control metrics were analyzed from raw data files and a survey about routine procedures. Results showed large differences in coverages, but observed variant allele frequencies in raw data files were in line with predefined variant allele frequencies. Many false negative results were found because of low-quality regions, which were not reported as such. It is recommended to disclose the reportable range, the fraction of targeted genomic regions for which calls of acceptable quality can be generated, to avoid any errors in therapy decisions. NGS can be a reliable technique, only if essential quality control during analysis is applied and reported. In the past, to test a single gene, several PCR-based platforms or Sanger sequencing techniques were used for testing predictive and prognostic markers in cancer.1McCourt C.M. McArt D.G. Mills K. Catherwood M.A. Maxwell P. Waugh D.J. Hamilton P. O'Sullivan J.M. Salto-Tellez M. Validation of next generation sequencing technologies in comparison to current diagnostic gold standards for BRAF, EGFR and KRAS mutational analysis.PLoS One. 2013; 8: e69604Crossref PubMed Scopus (91) Google Scholar, 2Martinez D.A. Nelson M.A. The next generation becomes the now generation.PLoS Genet. 2010; 6: e1000906Crossref PubMed Scopus (28) Google Scholar Currently, the testing of several genes and multiple variant hotspots has become common practice in cancer treatment decision making.3Cagle P.T. Raparia K. Portier B.P. Emerging biomarkers in personalized therapy of lung cancer.Adv Exp Med Biol. 2016; 890: 25-36Crossref PubMed Scopus (15) Google Scholar The recruitment of clinical trial patients is increasingly based on confirmed variants and the knowledge of the molecular tumor spectrum.4Renfro L.A. An M.W. Mandrekar S.J. Precision oncology: a new era of cancer clinical trials.Cancer Lett. 2017; 387: 121-126Abstract Full Text Full Text PDF PubMed Scopus (37) Google Scholar However, it is difficult to respect the turnaround time to test multiple genes sequentially by using Sanger sequencing. Moreover, the limited amount of available tumor tissue makes sequential analyses almost impossible. Thus, demand is increasing for methods that allow molecular testing of numerous variants simultaneously with low DNA input. Next-generation sequencing (NGS) can fulfill these requirements and is finding its way as primary technique for biomarker testing in tumor tissues in many laboratories.4Renfro L.A. An M.W. Mandrekar S.J. Precision oncology: a new era of cancer clinical trials.Cancer Lett. 2017; 387: 121-126Abstract Full Text Full Text PDF PubMed Scopus (37) Google Scholar, 5Liu J. Hu J. Cheng L. Ren W. Yang M. Liu B. Xie L. Qian X. Biomarkers predicting resistance to epidermal growth factor receptor-targeted therapy in metastatic colorectal cancer with wild-type KRAS.Onco Targets Ther. 2016; 9: 557-565Crossref PubMed Scopus (14) Google Scholar, 6Mu W. Lu H.M. Chen J. Li S. Elliott A.M. Sanger confirmation is required to achieve optimal sensitivity and specificity in next-generation sequencing panel testing.J Mol Diagn. 2016; 18: 923-932Abstract Full Text Full Text PDF PubMed Scopus (100) Google Scholar Different NGS technologies are available for performing whole-genome, whole-exome, or targeted sequencing analysis. Today, the latter is the preferred option for oncology biomarker testing. Whole-genome or whole-exome sequencing is too expensive for routine practice because the sequence depth should be high enough for analysis of tumor tissue, and there is a limited clinical actionability of most regions of the human genome.7Hagemann I.S. Cottrell C.E. Lockwood C.M. Design of targeted, capture-based, next generation sequencing tests for precision cancer therapy.Cancer Genet. 2013; 206: 420-431Abstract Full Text Full Text PDF PubMed Scopus (42) Google Scholar In addition, formalin-fixed, paraffin-embedded (FFPE) tissue that contains fragmented DNA is not yet optimized to be used for whole-genome sequencing.8Deans Z.C. Costa J.L. Cree I. Dequeker E. Edsjo A. Henderson S. Hummel M. Ligtenberg M.J. Loddo M. Machado J.C. Marchetti A. Marquis K. Mason J. Normanno N. Rouleau E. Schuuring E. Snelson K.M. Thunnissen E. Tops B. Williams G. van Krieken H. Hall J.A. IQN Path ASBLIntegration of next-generation sequencing in clinical diagnostic molecular pathology laboratories for analysis of solid tumours; an expert opinion on behalf of IQN Path ASBL.Virchows Arch. 2017; 470: 5-20Crossref PubMed Scopus (52) Google Scholar The NGS library preparation can be either PCR based or capture based and can be combined with different sequencing platforms.8Deans Z.C. Costa J.L. Cree I. Dequeker E. Edsjo A. Henderson S. Hummel M. Ligtenberg M.J. Loddo M. Machado J.C. Marchetti A. Marquis K. Mason J. Normanno N. Rouleau E. Schuuring E. Snelson K.M. Thunnissen E. Tops B. Williams G. van Krieken H. Hall J.A. IQN Path ASBLIntegration of next-generation sequencing in clinical diagnostic molecular pathology laboratories for analysis of solid tumours; an expert opinion on behalf of IQN Path ASBL.Virchows Arch. 2017; 470: 5-20Crossref PubMed Scopus (52) Google Scholar, 9Ross J.S. Cronin M. Whole cancer genome sequencing by next-generation methods.Am J Clin Pathol. 2011; 136: 527-539Crossref PubMed Scopus (130) Google Scholar, 10Jennings L.J. Arcila M.E. Corless C. Kamel-Reid S. Lubin I.M. Pfeifer J. Temple-Smolkin R.L. Voelkerding K.V. Nikiforova M.N. Guidelines for validation of next-generation sequencing-based oncology panels: a joint consensus recommendation of the Association for Molecular Pathology and College of American Pathologists.J Mol Diagn. 2017; 19: 341-365Abstract Full Text Full Text PDF PubMed Scopus (251) Google Scholar Sequencing costs per sample are decreasing for NGS (http://www.genome.gov/sequencingcosts, last accessed March 23, 2017). Although targeted sequencing was first performed for a few thousand dollars, this is now available for a few hundred dollars per sample.11Shao D. Lin Y. Liu J. Wan L. Liu Z. Cheng S. Fei L. Deng R. Wang J. Chen X. Liu L. Gu X. Liang W. He P. Wang J. Ye M. He J. A targeted next-generation sequencing method for identifying clinically relevant mutation profiles in lung adenocarcinoma.Sci Rep. 2016; 6: 22338Crossref PubMed Scopus (43) Google Scholar, 12Gonzalez-Garay M.L. The road from next-generation sequencing to personalized medicine.Per Med. 2014; 11: 523-544Crossref PubMed Scopus (22) Google Scholar The decreasing costs, the low turnaround time, the use of FFPE material, and the broad coverage of clinically relevant genes will further encourage the use of targeted NGS in routine practice of molecular pathology laboratories. The implementation of NGS also knows some limitations. It remains a challenge to handle the limited amount of DNA and the quality of DNA extracted from FFPE tissue in molecular pathology.13Betge J. Kerr G. Miersch T. Leible S. Erdmann G. Galata C.L. Zhan T. Gaiser T. Post S. Ebert M.P. Horisberger K. Boutros M. Amplicon sequencing of colorectal cancer: variant calling in frozen and formalin-fixed samples.PLoS One. 2015; 10: e0127146Crossref PubMed Scopus (26) Google Scholar In addition, the interpretation of the results with the use of bioinformatics becomes more complex. Many different quality control (QC) metrics can be applied to filter the huge amount of data and to determine the correct variants in the sample. Reporting the correct genotype of a tumor is especially important in decisions for targeted therapy. For example, confirmation of activating EGFR variants in non–small-cell lung cancers is essential before therapy with epidermal growth factor receptor tyrosine kinase inhibitors.14Lynch T.J. Bell D.W. Sordella R. Gurubhagavatula S. Okimoto R.A. Brannigan B.W. Harris P.L. Haserlat S.M. Supko J.G. Haluska F.G. Louis D.N. Christiani D.C. Settleman J. Haber D.A. Activating mutations in the epidermal growth factor receptor underlying responsiveness of non-small-cell lung cancer to gefitinib.N Engl J Med. 2004; 350: 2129-2139Crossref PubMed Scopus (9446) Google Scholar, 15Paez J.G. Janne P.A. Lee J.C. Tracy S. Greulich H. Gabriel S. Herman P. Kaye F.J. Lindeman N. Boggon T.J. Naoki K. Sasaki H. Fujii Y. Eck M.J. Sellers W.R. Johnson B.E. Meyerson M. EGFR mutations in lung cancer: correlation with clinical response to gefitinib therapy.Science. 2004; 304: 1497-1500Crossref PubMed Scopus (7986) Google Scholar Recommendations for using NGS in clinical practice describe various quality parameters that need to be taken into account; however, in most cases, no exact criteria for variant calling are given (www.wadsworth.org/sites/default/files/WebDoc/Updated%20NextGen%20Seq%20ONCO_Guidelines_032016.pdf, last accessed March 31, 2017).16Matthijs G. Souche E. Alders M. Corveleyn A. Eck S. Feenstra I. Race V. Sistermans E. Sturm M. Weiss M. Yntema H. Bakker E. Scheffer H. Bauer P. EuroGentest; European Society of Human GeneticsGuidelines for diagnostic next-generation sequencing.Eur J Hum Genet. 2016; 24: 2-5Crossref PubMed Scopus (271) Google Scholar, 17Gargis A.S. Kalman L. Berry M.W. Bick D.P. Dimmock D.P. Hambuch T. et al.Assuring the quality of next-generation sequencing in clinical laboratory practice.Nat Biotechnol. 2012; 30: 1033-1036Crossref PubMed Scopus (339) Google Scholar, 18Weiss M.M. Van der Zwaag B. Jongbloed J.D. Vogel M.J. Bruggenwirth H.T. Lekanne Deprez R.H. Mook O. Ruivenkamp C.A. van Slegtenhorst M.A. van den Wijngaard A. Waisfisz Q. Nelen M.R. van der Stoep N. Best practice guidelines for the use of next-generation sequencing applications in genome diagnostics: a national collaborative study of Dutch genome diagnostic laboratories.Hum Mutat. 2013; 34: 1313-1321Crossref PubMed Scopus (77) Google Scholar For instance, coverage, one of the most relevant technical variables in NGS, can help in troubleshooting errors and optimizing the laboratory's NGS workflow.10Jennings L.J. Arcila M.E. Corless C. Kamel-Reid S. Lubin I.M. Pfeifer J. Temple-Smolkin R.L. Voelkerding K.V. Nikiforova M.N. Guidelines for validation of next-generation sequencing-based oncology panels: a joint consensus recommendation of the Association for Molecular Pathology and College of American Pathologists.J Mol Diagn. 2017; 19: 341-365Abstract Full Text Full Text PDF PubMed Scopus (251) Google Scholar A tool to assure correct diagnostic results is participation in external quality assessment (EQA) schemes.19Organisation for Economic Co-operation and DevelopmentOECD Guidelines for Quality Assurance in Molecular Genetic Testing. OECD, Paris2007Google Scholar With participation in EQA, it can be verified whether the test method used and QC metrics used are valid and accurate, and reliable results are obtained. This study assessed the performance of NGS compared with other variant-detection assays in an EQA scheme for somatic variant analysis in the EGFR gene. However, a multigene analysis of DNA control material was performed by 26 laboratories that used nine different NGS technologies. The technical quality within and between the technologies was compared by re-analyzing the raw data files and the performance of the laboratories was evaluated. The study consisted of two parts. The first part was the analysis of patient FFPE samples for EGFR variants. The participants in this study were informed that the samples were part of the 2015 EGFR EQA scheme of the European Society of Pathology (ESP) (lung.eqascheme.org, accessed December 18, 2017). This EGFR EQA scheme was open to all laboratories worldwide. The scheme was organized according to international guidelines.20van Krieken J.H. Normanno N. Blackhall F. Boone E. Botti G. Carneiro F. Celik I. Ciardiello F. Cree I.A. Deans Z.C. Edsjo A. Groenen P.J. Kamarainen O. Kreipe H.H. Ligtenberg M.J. Marchetti A. Murray S. Opdam F.J. Patterson S.D. Patton S. Pinto C. Rouleau E. Schuuring E. Sterck S. Taron M. Tejpar S. Timens W. Thunnissen E. van de Ven P.M. Siebers A.G. Dequeker E. Guideline on the requirements of external quality assessment programs in molecular pathology.Virchows Arch. 2013; 462: 27-37Crossref PubMed Scopus (54) Google Scholar To determine the performance of a laboratory, scores were assigned by two independent assessors by comparison of the participant's results with the validated results. Two points could be obtained for each correctly genotyped sample for a total of 18 points, and points were deducted in case of a genotyping, technical, clerical, or nomenclature error. Genotyping errors were defined as false positive results, false negative results, or reporting of a wrong variant. A participant was seen as successful if at most one technical error and/or a nomenclature mistake or no more than one genotyping error without nomenclature mistake was made. Nine FFPE patient samples were evaluated in the scheme (Table 1). For three of the samples, mock clinical information was provided by the organization, and participants needed to submit written diagnostic reports with molecular results and interpretation of these cases.Table 1Overview of the Material Sent in Both Parts of the StudyVariablePart 1: samples of the 2015 EGFR EQA scheme of the ESPVariants in the samples (N = 9)EGFR wild-type (five samples)EGFR c.2156G>C p.(Gly719Ala)EGFR c.2573T>G p.(Leu858Arg)EGFR c.2235_2249del p.(Glu746_Ala750del)EGFR c.2369C>T p.(Thr790Met) and c.2573T>G p.(Leu858Arg)Part 2: sample APart 2: sample BMaterial sent1 μg75 μg or 300 μg, dependent on participant's panelVariants in sample>500 at different VAF>500 at different VAFVariants selected for the study16 variants in 10 genes299 variants in 20 genesPosition of the selected variantsExonicExonic, intronic, splice sitesRelevance of the selected variantsPathogenic (100%)Pathogenic (37.2%)Benign (8.64%)Uncertain (28.9%)Unknown (25.3%)Reference sequence EGFR: NM_005228.4. As the target regions for each NGS panel differ, the actual percentages varied for each technique.EQA, external quality assessment; ESP, European Society of Pathology; Sample A, Quantitative Multiplex DNA reference standard from Horizon Discovery; Sample B, Oncology Hotspot Control from Thermo Fisher Scientific; VAF, variant allele frequency. Open table in a new tab Reference sequence EGFR: NM_005228.4. As the target regions for each NGS panel differ, the actual percentages varied for each technique. EQA, external quality assessment; ESP, European Society of Pathology; Sample A, Quantitative Multiplex DNA reference standard from Horizon Discovery; Sample B, Oncology Hotspot Control from Thermo Fisher Scientific; VAF, variant allele frequency. For the selection of the participants of the second part of this study, an invitation was sent to more than 600 institutes from the ESP database. A candidature form was filed by 98 laboratories, and the final selection of 26 European laboratories was based on the used NGS methodology (panel and platform), accreditation status, NGS implementation date, and number of samples tested per year to obtain a group with enough variation on these aspects (Table 2).Table 2Overview of the Characteristics of the Participants in Both Parts of the StudyTechniqueNumber of NGS samples tested per yearAccredited laboratories, nNGS implementationNon–NGS-based commercial (N = 59)NAYes: 14NANo: 45Non–NGS-based laboratory developed (N = 22)NAYes: 12NANo: 10NGS-based commercial (N = 29)0–249: 12Yes: 16Before 2014: 4250–499: 5No: 132014–2015: 18>500: 62016: 1Missing: 6Missing: 6NGS-based laboratory developed (N = 12)0–249: 1Yes: 7Before 2014: 1250–499: 2No: 52014–2015: 2>500: 02016: 0Missing: 9Missing: 9For the accreditation status, different national and international standards were taken into account: ISO 15189 and ISO 17025 (International Organisation for Standardization) standards as recognized international accreditation standards, CAP 15189 (College of American Pathologists) as national accreditation standard, and widely used national standards such as the national standard in the Netherlands (CCKL). Missing data arose because these questions were only asked in the survey of part 2 (analysis of DNA control material).N, total number of laboratories in that group; NA, not applicable; NGS, next-generation sequencing. Open table in a new tab For the accreditation status, different national and international standards were taken into account: ISO 15189 and ISO 17025 (International Organisation for Standardization) standards as recognized international accreditation standards, CAP 15189 (College of American Pathologists) as national accreditation standard, and widely used national standards such as the national standard in the Netherlands (CCKL). Missing data arose because these questions were only asked in the survey of part 2 (analysis of DNA control material). N, total number of laboratories in that group; NA, not applicable; NGS, next-generation sequencing. The DNA control material originated from two different manufacturers: sample A was the Quantitative Multiplex DNA reference standard from Horizon Discovery (Cambridge, UK) and sample B was the Oncology Hotspot Control from Thermo Fisher Scientific (Waltham, MA) (Table 1). For this study, 20 genes were selected for which the participants needed to report results: AKT1, ALK, BRAF, CTNNB1, EGFR, ERBB2, ERBB4, FGFR2, FGFR3, KIT, KRAS, MAP2K1, MET, NRAS, PDGFRA, PIK3CA, PTEN, SMAD4, STK11, and TP53. This list was selected according to the overlap between the targeted regions covered by the different NGS methodologies, the clinical relevance of markers, and current availability of targeted therapies for biomarkers in these genes. The relevance (pathogenic versus benign) and position (intronic, exonic, splice site) of the variants in the selected genes was determined by the Biomedical Quality Assurance unit (KU Leuven, Leuven, Belgium) with the Ingenuity Variant Analysis software version 4.3.20170418 from Qiagen (Valencia, CA). The participants analyzed the DNA control material with their routine NGS methodology and were requested to report any variant above a variant allele frequency (VAF) of 1% in the 20 selected genes, regardless of the clinical relevance. A list with the identified variants and the raw data files [Binary Alignment Map (BAM) files] from the NGS analysis were submitted to the organizers of the study. In addition, questions were asked about the QC metrics used and the validation procedure of the participant's NGS technique. Not all questions in the survey were mandatory, and more than one option could be chosen for several questions; hence, the sum of percentages is not equal to 100%. After the participants submitted their results, all further analyses, discussed in Results, were performed at the Biomedical Quality Assurance Research Unit and the Center of Human Genetics of the KU Leuven. The limit of detection of the participant's technique was taken into account for the determination of correct or incorrect results. All reported variants in the DNA samples were cross-checked with the variant list provided by the manufacturer to identify possible false positive or false negative results. The BED files with target definitions were provided by the companies of the NGS panels. Technical information, such as total coverage and alternative allele coverage, was calculated from the provided BAM files with the use of bam-readcount software version 0.8 (https://github.com/genome/bam-readcount, last accessed May 2, 2017). A minimum mapping quality and base quality of 15 was applied. BED tools version 2.25.0 was used to analyze the observed coverages of participants.21Quinlan A.R. Hall I.M. BEDTools: a flexible suite of utilities for comparing genomic features.Bioinformatics. 2010; 26: 841-842Crossref PubMed Scopus (8774) Google Scholar, 22Quinlan A.R. BEDTools: the Swiss-Army tool for genome feature analysis.Curr Protoc Bioinformatics. 2014; 47: 11.2.1-11.2.34Crossref Scopus (571) Google Scholar The observed VAFs were calculated from this data set. Statistical analysis was done with IBM SPSS (New York, NY) statistics software version 22. Comparison of groups was done with a χ2 test or with Fisher's exact test in case the expected frequency was <5 in >20%. The results of the analysis of nine FFPE samples from the 2015 ESP EQA scheme for EGFR variant analysis were evaluated. In this scheme, 33 of 114 participants (28.9%) used their routine NGS technique for EGFR variant analysis. The average genotyping scores for laboratories that used NGS (N = 33, NGS laboratories) and laboratories that used another technique (N = 81, non-NGS laboratories) were 90.0% (16 of 18) and 87% (15.7 of 18), respectively. Only 79% (26 of 33) of the NGS laboratories and 64% (52 of 81) of the non-NGS laboratories made no genotyping errors (false positive results, false negative results, or reporting a wrong variant). More details on the genotyping errors can be found in Table 3. In addition, NGS laboratories tended to obtain the maximal score (18 of 18) more than non-NGS laboratories. However, the number of laboratories with a successful score was similar between both groups (Table 3). At samples level, the differences between the two groups were smaller, but still more genotyping errors occurred when analyzing the samples with a non-NGS technology (6%) than with an NGS technology (4%). Detailed results on sample level showed more false negative results than false positive results or reporting a wrong variant (Table 3). Of the 33 NGS laboratories, 12 used a laboratory-developed NGS panel. The average score of these 12 laboratories was 89% and 4 (33%) laboratories made a genotyping error. This lower average score was due to one laboratory that made four genotyping errors. More detailed information on the type of errors of the NGS laboratories versus the non-NGS laboratories can be found in Table 3. No significant differences could be observed between the two groups about the number of laboratories with major genotyping errors (P = 0.129), technical errors (P = 0.447), or a maximum performance score (18 of 18; P = 0.193) in the EQA scheme.23Keppens C. Tack V. Hart ‘t N. Tembuyser L. Ryska A. Pauwels P. Zwaenepoel K. Schuuring E. Cabillic F. Tornillo L. Warth A. Weichert W. Dequeker E. for the EQA assessors expert groupA stitch in time saves nine: external quality assessment rounds demonstrate improved quality of biomarker analysis in lung cancer.Oncotarget. 2018; 9: 20524-20538Crossref PubMed Scopus (19) Google ScholarTable 3Comparison of the Analysis Results between NGS Laboratories and Non-NGS Laboratories for the Patient Formalin-Fixed, Paraffin-Embedded SamplesNGS vs Non-NGS≥1 FP≥1 FN≥1 WM≥1 Technical failureMaximal score∗Maximal score is defined as a 100% score (18 of 18).Successful score†Successful score is defined as at most one test failure and/or a nomenclature mistake, or at most one genotype error without nomenclature mistake.Participating laboratories, n (%) NGS (N = 33)0 (0.0)5 (15.2)3 (9.1)10 (30.3)6 (18.2)17 (51.5) Non-NGS (N = 81)8 (9.9)23 (28.4)7 (8.6)19 (23.5)7 (8.6)42 (51.9)Samples _nalysed, n (%) NGS (N = 33)0 (0.0)9 (3.3)4 (1.3)15 (5.1)NANA Non-NGs (N = 81)10 (1.4)30 (4.1)7 (1.0)38 (5.2)NANAFN, false negative; FP, false positive; NA, not applicable; NGS, next-generation sequencing; WM, wrong mutation.∗ Maximal score is defined as a 100% score (18 of 18).† Successful score is defined as at most one test failure and/or a nomenclature mistake, or at most one genotype error without nomenclature mistake. Open table in a new tab FN, false negative; FP, false positive; NA, not applicable; NGS, next-generation sequencing; WM, wrong mutation. In addition to the genotypes for each sample, laboratories also submitted diagnostic reports. Only 3 of the 33 NGS laboratories (9.1%) reported which exons were not informative enough for a reliable conclusion, and one NGS laboratory reported that some exons were of suboptimal quality, but did not state the exact exons. None of these laboratories had a false positive or a false negative result in the analysis of the patient samples. Twenty-six institutes participated in the second part of the study, of which two institutes used two different NGS techniques (Table 4). Not all laboratories submitted the requested raw data files or the survey on QC metrics (Table 4). More than half of the participants were accredited (58%) according to a national or international standard. Half of them tested a limited number of samples with NGS in 2014 (0 to 249 samples). Approximately a quarter tested 250 to 499 samples in 2014 and another quarter tested >500 samples. Most laboratories (58%) worked both in a clinical trial setting and in a diagnostic setting (Table 2). All institutes used amplicon-based panels in combination with sequencing-by-synthesis (MiSeq; Illumina, San Diego, CA), pyrosequencing (GS Junior; Roche, Basel, Switzerland), or Ion Torrent semiconductor technology (PGM or Ion Proton; Thermo Fisher Scientific).Table 4Overview of the Participants and the Submitted Information Available for Further AnalysisIDPlatformPanelSample analyzedQC metric surveyRaw data submitted1MiSeq (Illumina)Tumor Actionable Mutations panel (Qiagen)A and BYesYes2A and BYesNo3A and BYesYes4aGS Junior (Roche)In-house panelA conclusiveB inconclusiveNoNA5A and BYesNA6A and BYesNA7A and BYesNA8PGM IonTorrent (Thermo Fisher Scientific)Ion AmpliSeq Colon and Lung Cancer Panel (Thermo Fisher Scientific)A conclusiveB inconclusiveYesYes, Sample A9A and BYesYes11A and BYesYes19A and BYesYes10Ion Proton (Thermo Fisher Scientific)A and BYesYes12PGM IonTorrent (Thermo Fisher Scientific)Oncomine Focus Assay (Thermo Fisher Scientific)A and BYesYes13PGM IonTorrent (Thermo Fisher Scientific)Oncomine Solid Tumor DNA kit (Thermo Fisher Scientific)A and BYesNo14A and BYesYes4bA and BYesNo15MiSeq (Illumina)Somatic 1 MASTR assay (Multiplicom)A and BYesYes16A and BYesNo17A and BYesNo18A conclusiveB no resultYesNo22MiSeq (Illumina)Tumor hotspot (Multiplicom)A and BYesYes24aMiSeq (Illumina)TruSeq Amplicon Cancer panel (Illumina)A and BYesYes25MiSeq dx (Illumina)A and BYesYes26MiSeq (Illumina)A and BYesYes23MiSeq (Illumina)TruSight tumor panel (Illumina)A and BYesYes24bA and BYesYes27A and BYesYes28A and BYesYesTwo institutes participated with two different techniques. Sample A: Quantitative Multiplex DNA reference standard from Horizon Discovery. Sample B: Oncology Hotspot Control from Thermo Fisher Scientific. Laboratories 4 and 14 and Laboratories 23 and 25 are identical. These laboratories participated in this study with two different NGS technologies.ID, identification; NA, not applicable; QC, quality control. Open table in a new tab Two institutes participated with two different techniques. Sample A: Quantitative Multiplex DNA reference standard from Horizon Discovery. Sample B: Oncology Hotspot Control from Thermo Fisher Scientific. Laboratories 4 and 14 and Laboratories 23 and 25 are identical. These laboratories participated in this study with two different NGS technologies. ID, identification; NA, not applicable; QC, quality control. To compare the technical performance of nine different NGS protocols, the raw data files (BAM files) were re-analyzed at a central laboratory. Hereby, two important QC metrics, the coverage and VAF of variants in the different panels, were studied in detail. Only the NGS panels used by at least two participants were taken into account. An overview of the average coverage of the amplicon panels for each technology and participant is given in Figure 1, A–D . To make sure