AI helps you reading Science

AI generates interpretation videos

AI extracts and analyses the key points of the paper to generate videos automatically


pub
Go Generating

AI Traceability

AI parses the academic lineage of this thesis


Master Reading Tree
Generate MRT

AI Insight

AI extracts a summary of this paper


Weibo:
To optimize the design of clinical trials, we introduce a novel Clinical Trial Result Prediction task

Predicting Clinical Trial Results by Implicit Evidence Integration

EMNLP 2020, pp.1461-1477, (2020)

Cited by: 0|Views196
Full Text
Bibtex
Weibo

Abstract

Clinical trials provide essential guidance for practicing Evidence-Based Medicine, though often accompanying with unendurable costs and risks. To optimize the design of clinical trials, we introduce a novel Clinical Trial Result Prediction (CTRP) task. In the CTRP framework, a model takes a PICO-formatted clinical trial proposal with its ...More

Code:

Data:

0
Introduction
  • Shall COVID-19 patients be treated with hydroxychloroquine? In the era of Evidence-Based Medicine (EBM, Sackett 1997), medical practice should be guided by well-designed and wellconducted clinical research, such as randomized controlled trials.
  • The World Health Organization (WHO) has launched a global megatrial, Solidarity (WHO, 2020), to prioritize clinical resources by recommending only four most promising therapies1
  • The rationale for this suggestion comes from the integration of evidence that they might be effective against coronaviruses or other related organisms in laboratory or clinical studies (Peymani et al, 2016; Sheahan et al, 2017; Morra et al, 2018).
Highlights
  • Shall COVID-19 patients be treated with hydroxychloroquine? In the era of Evidence-Based Medicine (EBM, Sackett 1997), medical practice should be guided by well-designed and wellconducted clinical research, such as randomized controlled trials
  • Medical Subject Headings (MeSH) Ontology: Since no external knowledge base (KB) is available for our task, we use the training set as an internal alternative: we map the I, C and O of the test
  • The open-domain question answering (QA) baselines perform even worse: for the MeSH Ontology method, the internal KB of only 8k entries is far from complete; for the Retrieval + Evidence Inference method, the PICO queries are so specific that no exactly relevant evidence can be found in other trials and retrieving only a few trials has limited utilities
  • BioBERT has about twice as much (5.1% v.s. 2.7%) |∆| in the adversarial setting as EBM-Net does. It suggests that EBM-Net is more robust to adversarial attacks, which is a vital property for healthcare applications
  • EBM-Net without adversarial pre-training is less robust than EBM-Net as well (3.0% v.s. 2.7%), but not as vulnerable as BioBERT, indicating that robustness can be learned by pre-training with original implicit evidence to some extent and further consolidated by the adversarial evidence
  • We introduce a novel task, Clinical Trial Result Prediction (CTRP), to predict clinical trial results without doing them
Methods
  • The authors compare to a variety of methods, ranging from trivial ones like Random and Majority to the stateof-the-art BioBERT model.
  • Random: the authors report the expected performance of randomly predicting the result for each instance.
  • Majority: the authors report the performance of predicting the majority class (→) for all test instances.n. Bag-of-Words + Logistic Regression: the authors concatenate the TF-IDF weighted bag-of-word vectors of B, P, I, C and O as features and use logistic regression for learning.
  • MeSH is a controlled and hierarchically-organized vocabulary for describing biomedical topics.
  • The authors find their nearest labeled instances in the training set, where the distance is defined by: d(i, j) =
Results
  • BioBERT has about twice as much (5.1% v.s. 2.7%) |∆| in the adversarial setting as EBM-Net does
  • It suggests that EBM-Net is more robust to adversarial attacks, which is a vital property for healthcare applications.
  • EBM-Net without adversarial pre-training is less robust than EBM-Net as well (3.0% v.s. 2.7%), but not as vulnerable as BioBERT, indicating that robustness can be learned by pre-training with original implicit evidence to some extent and further consolidated by the adversarial evidence
Conclusion
  • The authors introduce a novel task, CTRP, to predict clinical trial results without doing them.
  • Instead of using structured evidence that is prohibitively expensive to annotate, the authors heuristically collect 12M unstructured sentences as implicit evidence, and use large-scale CLM pretraining to learn the conditional ordering function required for solving the CTRP task.
  • The authors' EBM-Net model outperforms other strong baselines on the Evidence Integration dataset and is validated on COVID-19 clinical trials
Tables
  • Table1: Several examples of implicit evidence. Red, violet and blue denote superiority, equality and inferiority
  • Table2: Main results on the benchmark Evidence Integration dataset. |∆| denotes the absolute value of relative accuracy decrease from the standard to the adversarial setting. All numbers are percentages. (w/o: without)
Download tables as Excel
Related work
  • Predicting Clinical Trial Results: Most relevant works typically use only specific types or sources of information for prediction (e.g.: chemical structures (Gayvert et al, 2016), drug dosages or routes (Holford et al, 2000, 2010)). Gayvert et al (2016) predicts clinical trial results based on chemical properties of the candidate drugs. Clinical trial simulation (Holford et al, 2000, 2010) applies pharmacological models to predict the results of a specific intervention with different procedural factors, such as doses and sampling intervals. Some use closely related report information, e.g.: interim analyses (Broglio et al, 2014) or phase II data for just phase II trials (De Ridder, 2005). Our task is (1) more generalizable, since all potential PICO elements can be represented by free-texts and thus modeled in our work; and (2) aimed at evaluating new clinical trial proposals.

    Explicit Evidence Integration: It depends on the existence of structured evidence, i.e.: {P, I, C, O, R} (Wallace, 2019). Consequently, collecting such explicit evidence is vital for further analyses, and is also the objective for most relevant works: Some seek to find relevant papers through retrieval (Lee and Sun, 2018); many works are aimed at extracting PICO elements from published literature (Wallace et al, 2016; Singh et al, 2017; Marshall et al, 2017; Jin and Szolovits, 2018; Nye et al, 2018; Zhang et al, 2020); the evidence inference task extracts R for a given ICO query using the corresponding clinical trial report (Lehman et al, 2019; DeYoung et al, 2020). However, since getting expert annotations is expensive, these works are typically limited in scale, with only thousands of labeled instances. Few works have been done to utilize the automatically collected structured data for analyses. In this paper, we adopt an end-to-end approach, where we use large-scale pre-training to implicitly learn from free-text clinical evidence.
Funding
  • Out of the 373 mistakes EBM-Net makes on the test set, significantly less (11.8%, p<0.001 by permutation test) predictions are opposite to the ground-truth (e.g.: predicting ↑ when the label is ↓), also suggesting that EBM-Net effectively learn the relationship between comparison results
Reference
  • Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3606– 3611.
    Google ScholarLocate open access versionFindings
  • Kristine R Broglio, David N Stivers, and Donald A Berry. 2014. Predicting clinical trial results based on announcements of interim analyses. Trials, 15(1):73.
    Google ScholarLocate open access versionFindings
  • Filip De Ridder. 2005. Predicting the outcome of phase iii trials using phase ii data: a case study of clinical trial simulation in late stage drug development. Basic & clinical pharmacology & toxicology, 96(3):235–241.
    Google ScholarLocate open access versionFindings
  • Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Jay DeYoung, Eric Lehman, Ben Nye, Iain J. Marshall, and Byron C. Wallace. 2020. Evidence inference 2.0: More data, better models.
    Google ScholarFindings
  • Kaitlyn M Gayvert, Neel S Madhukar, and Olivier Elemento. 201A data-driven approach to predicting successes and failures of clinical trials. Cell chemical biology, 23(10):1294–1301.
    Google ScholarLocate open access versionFindings
  • N Holford, SC Ma, and BA Ploeger. 2010. Clinical trial simulation: a review. Clinical Pharmacology & Therapeutics, 88(2):166–182.
    Google ScholarLocate open access versionFindings
  • N. H. G. Holford, H. C. Kimko, J. P. R. Monteleone, and C. C. Peck. 2000. Simulation of clinical trials. Annual Review of Pharmacology and Toxicology, 40(1):209–234. PMID: 10836134.
    Locate open access versionFindings
  • Xiaoli Huang, Jimmy Lin, and Dina Demner-Fushman. 2006. Evaluation of pico as a knowledge representation for clinical questions. In AMIA annual symposium proceedings, volume 2006, page 35American Medical Informatics Association.
    Google ScholarLocate open access versionFindings
  • Di Jin and Peter Szolovits. 2018. PICO element detection in medical text via long short-term memory neural networks. In Proceedings of the BioNLP 2018 workshop, pages 67–75, Melbourne, Australia. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Nitin Jindal and Bing Liu. 2006. Mining comparative sentences and relations. In Proceedings of the 21st National Conference on Artificial Intelligence - Volume 2, AAAI’06, page 1331–1336. AAAI Press.
    Google ScholarLocate open access versionFindings
  • Christopher Kennedy. 2004.
    Google ScholarFindings
  • Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
    Findings
  • Grace E. Lee and Aixin Sun. 2018. Seed-driven document ranking for systematic reviews in evidencebased medicine. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR ’18, page 455–464, New York, NY, USA. Association for Computing Machinery.
    Google ScholarLocate open access versionFindings
  • Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240.
    Google ScholarLocate open access versionFindings
  • Eric Lehman, Jay DeYoung, Regina Barzilay, and Byron C. Wallace. 2019. Inferring which medical treatments work from reports of clinical trials. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3705–3717, Minneapolis, Minnesota. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579–2605.
    Google ScholarLocate open access versionFindings
  • Iain Marshall, Joel Kuiper, Edward Banner, and Byron C. Wallace. 2017. Automating biomedical evidence synthesis: RobotReviewer. In Proceedings of ACL 2017, System Demonstrations, pages 7–12, Vancouver, Canada. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Mandeep R Mehra, Sapan S Desai, Frank Ruschitzka, and Amit N Patel. 2020. Hydroxychloroquine or chloroquine with or without a macrolide for treatment of covid-19: a multinational registry analysis. The Lancet.
    Google ScholarLocate open access versionFindings
  • Pasquale Minervini and Sebastian Riedel. 2018. Adversarially regularising neural nli models to integrate logical background knowledge. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 65–74.
    Google ScholarLocate open access versionFindings
  • Mostafa Ebraheem Morra, Le Van Thanh, Mohamed Gomaa Kamel, Ahmed Abdelmotaleb Ghazy, Ahmed MA Altibi, Lu Minh Dat, Tran Ngoc Xuan Thy, Nguyen Lam Vuong, Mostafa Reda Mostafa, Sarah Ibrahim Ahmed, et al. 2018. Clinical outcomes of current medical approaches for middle east respiratory syndrome: A systematic review and meta-analysis. Reviews in medical virology, 28(3):e1977.
    Google ScholarLocate open access versionFindings
  • Benjamin Nye, Junyi Jessy Li, Roma Patel, Yinfei Yang, Iain Marshall, Ani Nenkova, and Byron Wallace. 2018. A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 197–207, Melbourne, Australia. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc.
    Google ScholarLocate open access versionFindings
  • P Peymani, S Ghavami, B Yeganeh, R Tabrizi, S Sabour, B Geramizadeh, MR Fattahi, SM Ahmadi, and KB Lankarani. 2016. Effect of chloroquine on some clinical and biochemical parameters in nonresponse chronic hepatitis c virus infection patients: pilot clinical trial. Acta bio-medica: Atenei Parmensis, 87(1):46.
    Google ScholarLocate open access versionFindings
  • David L Sackett. 1997. Evidence-based medicine. In Seminars in perinatology, pages 3–5. Elsevier.
    Google ScholarLocate open access versionFindings
  • Timothy P Sheahan, Amy C Sims, Rachel L Graham, Vineet D Menachery, Lisa E Gralinski, James B Case, Sarah R Leist, Krzysztof Pyrc, Joy Y Feng, Iva Trantcheva, et al. 2017. Broad-spectrum antiviral gs-5734 inhibits both epidemic and zoonotic coronaviruses. Science translational medicine, 9(396).
    Google ScholarLocate open access versionFindings
  • Gaurav Singh, Iain J. Marshall, James Thomas, John Shawe-Taylor, and Byron C. Wallace. 2017. A neural candidate-selector architecture for automatic structured clinical text annotation. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM ’17, page 1519–1528, New York, NY, USA. Association for Computing Machinery.
    Google ScholarLocate open access versionFindings
  • Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc.
    Google ScholarLocate open access versionFindings
  • Byron C. Wallace. 2019. What does the evidence say? models to help make sense of the biomedical literature. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 6416–6420. International Joint Conferences on Artificial Intelligence Organization.
    Google ScholarLocate open access versionFindings
  • Byron C. Wallace, Joel Kuiper, Aakash Sharma, Mingxi (Brian) Zhu, and Iain J. Marshall. 2016. Extracting pico sentences from clinical trial reports using supervised distant supervision. Journal of Machine Learning Research, 17(132):1–25.
    Google ScholarLocate open access versionFindings
  • Haohan Wang, Da Sun, and Eric P Xing. 2019. What if we simply swap the two text fragments? a straightforward yet effective way to test the robustness of methods to confounding signals in nature language inference tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7136–7143.
    Google ScholarLocate open access versionFindings
  • Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Darrin Eide, Kathryn Funk, Rodney Kinney, Ziyang Liu, William Merrill, et al. 2020a. Cord-19: The covid-19 open research dataset. arXiv preprint arXiv:2004.10706.
    Findings
  • Yeming Wang, Dingyu Zhang, Guanhua Du, Ronghui Du, Jianping Zhao, Yang Jin, Shouzhi Fu, Ling Gao, Zhenshun Cheng, Qiaofa Lu, et al. 2020b. Remdesivir in adults with severe covid-19: a randomised, double-blind, placebo-controlled, multicentre trial. The Lancet.
    Google ScholarFindings
  • WHO. 2020. Solidarity clinical trial for covid-19 treatments.
    Google ScholarFindings
  • Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
    Findings
  • Chi Heem Wong, Kien Wei Siah, and Andrew W Lo. 2019. Estimation of clinical trial success rates and related parameters. Biostatistics, 20(2):273–286.
    Google ScholarLocate open access versionFindings
  • Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019. End-to-end open-domain question answering with BERTserini. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 72–77, Minneapolis, Minnesota. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Tengteng Zhang, Yiqin Yu, Jing Mei, Zefang Tang, Xiang Zhang, and Shaochun Li. 2020. Unlocking the power of deep pico extraction: Step-wise medical ner identification. arXiv preprint arXiv:2005.06601.
    Findings
Author
Qiao Jin
Qiao Jin
Chuanqi Tan
Chuanqi Tan
Mosha Chen
Mosha Chen
Xiaozhong Liu
Xiaozhong Liu
Songfang Huang
Songfang Huang
Your rating :
0

 

Tags
Comments
数据免责声明
页面数据均来自互联网公开来源、合作出版商和通过AI技术自动分析结果,我们不对页面数据的有效性、准确性、正确性、可靠性、完整性和及时性做出任何承诺和保证。若有疑问,可以通过电子邮件方式联系我们:report@aminer.cn
小科