AI helps you reading Science

AI generates interpretation videos

AI extracts and analyses the key points of the paper to generate videos automatically


pub
Go Generating

AI Traceability

AI parses the academic lineage of this thesis


Master Reading Tree
Generate MRT

AI Insight

AI extracts a summary of this paper


Weibo:
We present an analysis of different word embedding methods and investigate their effectiveness on four clinical concept extraction tasks

Enhancing Clinical Concept Extraction with Contextual Embedding.

JOURNAL OF THE AMERICAN MEDICAL INFORMATICS ASSOCIATION, no. 11 (2019): 1297-1304

Cited: 30|Views155
EI WOS

Abstract

Objective: Neural network-based representations ("embeddings") have dramatically advanced natural language processing (NLP) tasks, including clinical NLP tasks such as concept extraction. Recently, however, more advanced embedding methods and representations (eg, ELMo, BERT) have further pushed the state of the art in NLP, yet there are n...More

Code:

Data:

0
Introduction
Highlights
  • Concept extraction is the most common clinical natural language processing (NLP) task (Tang et al, 2013; Kundeti et al, 2016; Unanue et al, 2017; Wang et al, 2018b), and a precursor to downstream tasks such as relations (Rink et al, 2011), frame parsing (Gupta et al, 2018; Si and Roberts, 2018), co-reference (Lee et al, 2011), and phenotyping (Xu et al, 2011; Velupillai et al, 2018)
  • We aim to explore the potential impact these representations have on clinical concept extraction
  • A performance increase for clinical concept extraction that achieves state-of-the-art results on all four corpora
  • The performance on the respective test sets for the embedding methods on the four clinical concept extraction tasks are reported in Table 3
  • We present an analysis of different word embedding methods and investigate their effectiveness on four clinical concept extraction tasks
  • The efficacy of contextual embeddings over traditional word vector representations are highlighted by comparing the performances on clinical concept extraction
Methods
  • The authors consider both off-the-shelf embeddings from the open domain as well as pretraining clinical domain embeddings on clinical notes from MIMIC-III (Johnson et al, 2016), which is a public database of Intensive Care Unit (ICU) patients.

    For the traditional word-embedding experiments, the static embeddings are fed into a BiLSTM CRF architecture.
  • For ELMo, the context-independent embeddings with trainable weights are used to form context-dependent embeddings, which are fed into the downstream task.
  • The context-dependent embedding is obtained through a low-dimensional projection and a highway connection after a stacked layer of a character-based Convolutional Neural Network and a two-layer Bi-LSTM language model.
  • The contextual word embedding is formed with a trainable aggregation of highly-connected bi-LM.
  • The contextual word embedding for each word is fed into the prior state-of-the-art sequence labeling architecture, Bi-LSTM CRF, to predict the label for each token
Results
  • 5.1 Performance Comparison

    The performance on the respective test sets for the embedding methods on the four clinical concept extraction tasks are reported in Table 3.
  • For i2b2 2010, the best performance is achieved by BERTLARGE(MIMIC) with an F1 of 90.25.
  • It improves the performance by 5.18 over the best performance of the traditional embeddings achieved by GloVe (MIMIC) with an F1 of 85.07.
  • As expected, both ELMo and BERT clinical embeddings outperform the off-the-shelf embeddings with relative increase up to 10%
Conclusion
  • This study explores the effects of numerous embedding methods on four clinical concept extraction tasks.
  • With the most advanced language model representation method pretrained on a large clinical corpus, namely BERT LARGE(MIMIC), the authors achieved new state-of-the-art performances across all tasks.
  • BERT LARGE(MIMIC) outperform the state-ofthe-art models on all four tasks with respective F-In this paper, the authors present an analysis of different word embedding methods and investigate their effectiveness on four clinical concept extraction tasks.
  • The efficacy of contextual embeddings over traditional word vector representations are highlighted by comparing the performances on clinical concept extraction.
  • The authors' results highlight the benefits of embeddings through unsupervised pretraining on clinical text corpora, which achieve higher performance than off-the-shelf embedding models and result in new state-of-the-art performance across all tasks
Tables
  • Table1: Descriptive statistics for concept extraction datasets
  • Table2: Resources of off-the-shelf embeddings from open domain. (*Vocabulary size calculated after word-piece tokenization)
  • Table3: Test set comparison in exact F1 of embedding methods across tasks. SOTA: state-of-the-art
  • Table4: Performance of each label category with pre-trained MIMIC models on i2b2 2010 task
  • Table5: Performance of each label category with pretrained MIMIC models on i2b2 2012 task
Download tables as Excel
Funding
  • This work was supported by the U.S National Institutes of Health (NIH) and the Cancer Prevention and Research Institute of Texas (CPRIT)
Reference
  • Martın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016. Tensorflow: a system for large-scale machine learning. In OSDI, volume 16, pages 265– 283.
    Google ScholarLocate open access versionFindings
  • Emily Alsentzer, John R Murphy, Willie Boag, WeiHung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical bert embeddings. arXiv preprint arXiv:1904.03323.
    Findings
  • Steven Bethard, Guergana Savova, Wei-Te Chen, Leon Derczynski, James Pustejovsky, and Marc Verhagen. 2016. Semeval-2016 task 12: Clinical tempeval. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 1052– 1062.
    Google ScholarLocate open access versionFindings
  • Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606.
    Findings
  • Allen C Browne, Alexa T McCray, and Suresh Srinivasan. 2000. The specialist lexicon. National Library of Medicine Technical Reports, pages 18–21.
    Google ScholarLocate open access versionFindings
  • Raghavendra Chalapathy, Ehsan Zare Borzeshi, and Massimo Piccardi. 201Bidirectional lstm-crf for clinical concept extraction. arXiv preprint arXiv:1611.08373.
    Findings
  • FX Chang, J Guo, WR Xu, and S Relly Chung. 2015. Application of word embeddings in biomedical named entity recognition tasks. Journal of Digital Information Management, 13(5).
    Google ScholarLocate open access versionFindings
  • Berry De Bruijn, Colin Cherry, Svetlana Kiritchenko, Joel Martin, and Xiaodan Zhu. 2011. Machinelearned solutions for three stages of clinical information extraction: the state of the art at i2b2 2010. Journal of the American Medical Informatics Association, 18(5):557–562.
    Google ScholarLocate open access versionFindings
  • Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
    Findings
  • Noemie Elhadad, Sameer Pradhan, Sharon Gorman, Suresh Manandhar, Wendy Chapman, and Guergana Savova. 2015. Semeval-2015 task 14: Analysis of clinical text. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 303–310.
    Google ScholarLocate open access versionFindings
  • Edson Florez, Frederic Precioso, Michel Riveill, and Romaric Pighetti. 2018. Named entity recognition using neural networks for clinical notes. In International Workshop on Medication and Adverse Drug Event Detection, pages 7–15.
    Google ScholarLocate open access versionFindings
  • Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256.
    Google ScholarLocate open access versionFindings
  • Anupama Gupta, Imon Banerjee, and Daniel L Rubin. 2018. Automatic information extraction from unstructured mammography reports using distributed semantics. Journal of biomedical informatics, 78:78–86.
    Google ScholarLocate open access versionFindings
  • Maryam Habibi, Leon Weber, Mariana Neves, David Luis Wiegandt, and Ulf Leser. 2017. Deep learning with word embeddings improves biomedical named entity recognition. Bioinformatics, 33(14):i37–i48.
    Google ScholarLocate open access versionFindings
  • Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. 2019. Clinicalbert: Modeling clinical notes and predicting hospital readmission. arXiv preprint arXiv:1904.05342.
    Findings
  • Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 20Mimic-iii, a freely accessible critical care database. Scientific data, 3:160035.
    Google ScholarLocate open access versionFindings
  • Liadh Kelly, Lorraine Goeuriot, Hanna Suominen, Tobias Schreck, Gondy Leroy, Danielle L Mowery, Sumithra Velupillai, Wendy W Chapman, David Martinez, Guido Zuccon, et al. 2014. Overview of the share/clef ehealth evaluation lab 2014. In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 172– 191.
    Google ScholarLocate open access versionFindings
  • Srinivasa Rao Kundeti, J Vijayananda, Srikanth Mujjiga, and M Kalyan. 2016. Clinical named entity recognition: Challenges and opportunities. In Big Data (Big Data), 2016 IEEE International Conference on, pages 1937–1945.
    Google ScholarLocate open access versionFindings
  • Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. arXiv preprint arXiv:1603.01360.
    Findings
  • Heeyoung Lee, Yves Peirsman, Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2011. Stanford’s multi-pass sieve coreference resolution system at the conll-2011 shared task. In Proceedings of the fifteenth conference on computational natural language learning: Shared task, pages 28–34.
    Google ScholarLocate open access versionFindings
  • Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: pre-trained biomedical language representation model for biomedical text mining. arXiv preprint arXiv:1901.08746.
    Findings
  • Zengjian Liu, Ming Yang, Xiaolong Wang, Qingcai Chen, Buzhou Tang, Zhe Wang, and Hua Xu. 2017. Entity recognition from clinical texts via recurrent neural network. BMC medical informatics and decision making, 17(2):67.
    Google ScholarLocate open access versionFindings
  • Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119.
    Google ScholarLocate open access versionFindings
  • Serguei Pakhomov, Bridget McInnes, Terrence Adam, Ying Liu, Ted Pedersen, and Genevieve B Melton. 2010. Semantic similarity and relatedness between clinical terms: an experimental study. In AMIA annual symposium proceedings, volume 2010, page 572. American Medical Informatics Association.
    Google ScholarLocate open access versionFindings
  • Serguei VS Pakhomov, Greg Finley, Reed McEwan, Yan Wang, and Genevieve B Melton. 2016. Corpus domain effects on distributional semantic modeling of medical terms. Bioinformatics, 32(23):3635– 3644.
    Google ScholarLocate open access versionFindings
  • Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543.
    Google ScholarLocate open access versionFindings
  • Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365.
    Findings
  • Sameer Pradhan, Noemie Elhadad, Wendy Chapman, Suresh Manandhar, and Guergana Savova. 2014. Semeval-2014 task 7: Analysis of clinical text. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 54–62.
    Google ScholarLocate open access versionFindings
  • Bryan Rink, Sanda Harabagiu, and Kirk Roberts. 2011. Automatic extraction of relations between medical concepts in clinical texts. Journal of the American Medical Informatics Association, 18(5):594–600.
    Google ScholarLocate open access versionFindings
  • Kirk Roberts. 2016. Assessing the corpus size vs. similarity trade-off for word embeddings in clinical nlp. In Proceedings of the Clinical Natural Language Processing Workshop (ClinicalNLP), pages 54–63.
    Google ScholarLocate open access versionFindings
  • Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149–5152. IEEE.
    Google ScholarLocate open access versionFindings
  • Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603.
    Findings
  • Yikang Shen, Wenge Rong, Nan Jiang, Baolin Peng, Jie Tang, and Zhang Xiong. 2017. Word embedding based correlation model for question/answer matching. In AAAI, pages 3511–3517.
    Google ScholarLocate open access versionFindings
  • Yuqi Si and Kirk Roberts. 2018. A Frame-Based NLP System for Cancer-Related Information Extraction. In AMIA Annual Symposium Proceedings, volume 2018, page 1524. American Medical Informatics Association.
    Google ScholarLocate open access versionFindings
  • Amber Stubbs, Christopher Kotfila, and Ozlem Uzuner. 2015. Automated systems for the de-identification of longitudinal clinical narratives: Overview of 2014 i2b2/uthealth shared task track 1. Journal of biomedical informatics, 58:S11–S19.
    Google ScholarLocate open access versionFindings
  • Weiyi Sun, Anna Rumshisky, and Ozlem Uzuner. 2013. Evaluating temporal relations in clinical text: 2012 i2b2 challenge. Journal of the American Medical Informatics Association, 20(5):806–813.
    Google ScholarLocate open access versionFindings
  • Hanna Suominen, Sanna Salantera, Sumithra Velupillai, Wendy W Chapman, Guergana Savova, Noemie Elhadad, Sameer Pradhan, Brett R South, Danielle L Mowery, Gareth JF Jones, et al. 2013. Overview of the share/clef ehealth evaluation lab 2013. In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 212– 231.
    Google ScholarLocate open access versionFindings
  • Buzhou Tang, Hongxin Cao, Yonghui Wu, Min Jiang, and Hua Xu. 2013. Recognizing clinical entities in hospital discharge summaries using structural support vector machines with word representation features. BMC Medical Informatics and Decision Making, 13(1):S1.
    Google ScholarLocate open access versionFindings
  • Buzhou Tang, Qingcai Chen, Xiaolong Wang, Yonghui Wu, Yaoyun Zhang, Min Jiang, Jingqi Wang, and Hua Xu. 2015. Recognizing disjoint clinical concepts in clinical text using machine learning-based methods. In AMIA Annual Symposium Proceedings, volume 2015, page 1184.
    Google ScholarLocate open access versionFindings
  • Inigo Jauregi Unanue, Ehsan Zare Borzeshi, and Massimo Piccardi. 2017. Recurrent neural networks with specialized word embeddings for healthdomain named-entity recognition. Journal of biomedical informatics, 76:102–109.
    Google ScholarLocate open access versionFindings
  • Ozlem Uzuner, Brett R South, Shuying Shen, and Scott L DuVall. 2011. 2010 i2b2/va challenge on concepts, assertions, and relations in clinical text. Journal of the American Medical Informatics Association, 18(5):552–556.
    Google ScholarLocate open access versionFindings
  • Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008.
    Google ScholarLocate open access versionFindings
  • Sumithra Velupillai, Hanna Suominen, Maria Liakata, Angus Roberts, Anoop D Shah, Katherine Morley, David Osborn, Joseph Hayes, Robert Stewart, Johnny Downs, et al. 2018. Using clinical natural language processing for health outcomes research: Overview and actionable suggestions for future advances. Journal of biomedical informatics, 88:11– 19.
    Google ScholarLocate open access versionFindings
  • Yanshan Wang, Sijia Liu, Naveed Afzal, Majid Rastegar-Mojarad, Liwei Wang, Feichen Shen, Paul Kingsbury, and Hongfang Liu. 2018a. A comparison of word embeddings for the biomedical natural language processing. Journal of biomedical informatics, 87:12–20.
    Google ScholarLocate open access versionFindings
  • Yanshan Wang, Liwei Wang, Majid Rastegar-Mojarad, Sungrim Moon, Feichen Shen, Naveed Afzal, Sijia Liu, Yuqun Zeng, Saeed Mehrabi, Sunghwan Sohn, et al. 2018b. Clinical information extraction applications: a literature review. Journal of biomedical informatics, 77:34–49.
    Google ScholarLocate open access versionFindings
  • Yonghui Wu, Jun Xu, Min Jiang, Yaoyun Zhang, and Hua Xu. 2015. A study of neural word embeddings for named entity recognition in clinical text. In AMIA Annual Symposium Proceedings, volume 2015, page 1326.
    Google ScholarLocate open access versionFindings
  • Hua Xu, Zhenming Fu, Anushi Shah, Yukun Chen, Neeraja B Peterson, Qingxia Chen, Subramani Mani, Mia A Levy, Qi Dai, and Josh C Denny. 2011. Extracting and integrating data from entire electronic health records for detecting colorectal cancer cases. In AMIA Annual Symposium Proceedings, volume 2011, page 1564.
    Google ScholarLocate open access versionFindings
  • Yaoyun Zhang, Jingqi Wang, Buzhou Tang, Yonghui Wu, Min Jiang, Yukun Chen, and Hua Xu. 2014. UTH CCB: a report for semeval 2014–task 7 analysis of clinical text. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 802–806.
    Google ScholarLocate open access versionFindings
  • Henghui Zhu, Ioannis Ch Paschalidis, and Amir Tahmasebi. 2018. Clinical concept extraction with contextual word embedding. arXiv preprint arXiv:1810.10566.
    Findings
Author
0
Your rating :

No Ratings

Tags
Comments
数据免责声明
页面数据均来自互联网公开来源、合作出版商和通过AI技术自动分析结果,我们不对页面数据的有效性、准确性、正确性、可靠性、完整性和及时性做出任何承诺和保证。若有疑问,可以通过电子邮件方式联系我们:report@aminer.cn