AI helps you reading Science
AI generates interpretation videos
AI extracts and analyses the key points of the paper to generate videos automatically
AI parses the academic lineage of this thesis
AI extracts a summary of this paper
Enhancing Clinical Concept Extraction with Contextual Embedding.
JOURNAL OF THE AMERICAN MEDICAL INFORMATICS ASSOCIATION, no. 11 (2019): 1297-1304
- Concept extraction is the most common clinical natural language processing (NLP) task (Tang et al, 2013; Kundeti et al, 2016; Unanue et al, 2017; Wang et al, 2018b), and a precursor to downstream tasks such as relations (Rink et al, 2011), frame parsing (Gupta et al, 2018; Si and Roberts, 2018), co-reference (Lee et al, 2011), and phenotyping (Xu et al, 2011; Velupillai et al, 2018).
- Recent advances in contextualized representations, including ELMo (Peters et al, 2018) and BERT (Devlin et al, 2018), have pushed performance even further
- These have demonstrated that relatively simple downstream models using contextualized embeddings can outperform complex models (Seo et al, 2016) using embeddings such as word2vec (Mikolov et al, 2013) and GloVe (Pennington et al, 2014).
- This approach enables handling of infrequent words that are not present in the training vocabulary, alleviating some out-ofvocabulary issues
- Concept extraction is the most common clinical natural language processing (NLP) task (Tang et al, 2013; Kundeti et al, 2016; Unanue et al, 2017; Wang et al, 2018b), and a precursor to downstream tasks such as relations (Rink et al, 2011), frame parsing (Gupta et al, 2018; Si and Roberts, 2018), co-reference (Lee et al, 2011), and phenotyping (Xu et al, 2011; Velupillai et al, 2018)
- We aim to explore the potential impact these representations have on clinical concept extraction
- A performance increase for clinical concept extraction that achieves state-of-the-art results on all four corpora
- The performance on the respective test sets for the embedding methods on the four clinical concept extraction tasks are reported in Table 3
- We present an analysis of different word embedding methods and investigate their effectiveness on four clinical concept extraction tasks
- The efficacy of contextual embeddings over traditional word vector representations are highlighted by comparing the performances on clinical concept extraction
- The authors consider both off-the-shelf embeddings from the open domain as well as pretraining clinical domain embeddings on clinical notes from MIMIC-III (Johnson et al, 2016), which is a public database of Intensive Care Unit (ICU) patients.
For the traditional word-embedding experiments, the static embeddings are fed into a BiLSTM CRF architecture.
- For ELMo, the context-independent embeddings with trainable weights are used to form context-dependent embeddings, which are fed into the downstream task.
- The context-dependent embedding is obtained through a low-dimensional projection and a highway connection after a stacked layer of a character-based Convolutional Neural Network and a two-layer Bi-LSTM language model.
- The contextual word embedding is formed with a trainable aggregation of highly-connected bi-LM.
- The contextual word embedding for each word is fed into the prior state-of-the-art sequence labeling architecture, Bi-LSTM CRF, to predict the label for each token
- 5.1 Performance Comparison
The performance on the respective test sets for the embedding methods on the four clinical concept extraction tasks are reported in Table 3.
- For i2b2 2010, the best performance is achieved by BERTLARGE(MIMIC) with an F1 of 90.25.
- It improves the performance by 5.18 over the best performance of the traditional embeddings achieved by GloVe (MIMIC) with an F1 of 85.07.
- As expected, both ELMo and BERT clinical embeddings outperform the off-the-shelf embeddings with relative increase up to 10%
- This study explores the effects of numerous embedding methods on four clinical concept extraction tasks.
- With the most advanced language model representation method pretrained on a large clinical corpus, namely BERT LARGE(MIMIC), the authors achieved new state-of-the-art performances across all tasks.
- BERT LARGE(MIMIC) outperform the state-ofthe-art models on all four tasks with respective F-In this paper, the authors present an analysis of different word embedding methods and investigate their effectiveness on four clinical concept extraction tasks.
- The efficacy of contextual embeddings over traditional word vector representations are highlighted by comparing the performances on clinical concept extraction.
- The authors' results highlight the benefits of embeddings through unsupervised pretraining on clinical text corpora, which achieve higher performance than off-the-shelf embedding models and result in new state-of-the-art performance across all tasks
- Table1: Descriptive statistics for concept extraction datasets
- Table2: Resources of off-the-shelf embeddings from open domain. (*Vocabulary size calculated after word-piece tokenization)
- Table3: Test set comparison in exact F1 of embedding methods across tasks. SOTA: state-of-the-art
- Table4: Performance of each label category with pre-trained MIMIC models on i2b2 2010 task
- Table5: Performance of each label category with pretrained MIMIC models on i2b2 2012 task
- This work was supported by the U.S National Institutes of Health (NIH) and the Cancer Prevention and Research Institute of Texas (CPRIT)
- Martın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016. Tensorflow: a system for large-scale machine learning. In OSDI, volume 16, pages 265– 283.
- Emily Alsentzer, John R Murphy, Willie Boag, WeiHung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical bert embeddings. arXiv preprint arXiv:1904.03323.
- Steven Bethard, Guergana Savova, Wei-Te Chen, Leon Derczynski, James Pustejovsky, and Marc Verhagen. 2016. Semeval-2016 task 12: Clinical tempeval. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 1052– 1062.
- Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606.
- Allen C Browne, Alexa T McCray, and Suresh Srinivasan. 2000. The specialist lexicon. National Library of Medicine Technical Reports, pages 18–21.
- Raghavendra Chalapathy, Ehsan Zare Borzeshi, and Massimo Piccardi. 201Bidirectional lstm-crf for clinical concept extraction. arXiv preprint arXiv:1611.08373.
- FX Chang, J Guo, WR Xu, and S Relly Chung. 2015. Application of word embeddings in biomedical named entity recognition tasks. Journal of Digital Information Management, 13(5).
- Berry De Bruijn, Colin Cherry, Svetlana Kiritchenko, Joel Martin, and Xiaodan Zhu. 2011. Machinelearned solutions for three stages of clinical information extraction: the state of the art at i2b2 2010. Journal of the American Medical Informatics Association, 18(5):557–562.
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
- Noemie Elhadad, Sameer Pradhan, Sharon Gorman, Suresh Manandhar, Wendy Chapman, and Guergana Savova. 2015. Semeval-2015 task 14: Analysis of clinical text. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 303–310.
- Edson Florez, Frederic Precioso, Michel Riveill, and Romaric Pighetti. 2018. Named entity recognition using neural networks for clinical notes. In International Workshop on Medication and Adverse Drug Event Detection, pages 7–15.
- Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256.
- Anupama Gupta, Imon Banerjee, and Daniel L Rubin. 2018. Automatic information extraction from unstructured mammography reports using distributed semantics. Journal of biomedical informatics, 78:78–86.
- Maryam Habibi, Leon Weber, Mariana Neves, David Luis Wiegandt, and Ulf Leser. 2017. Deep learning with word embeddings improves biomedical named entity recognition. Bioinformatics, 33(14):i37–i48.
- Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. 2019. Clinicalbert: Modeling clinical notes and predicting hospital readmission. arXiv preprint arXiv:1904.05342.
- Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 20Mimic-iii, a freely accessible critical care database. Scientific data, 3:160035.
- Liadh Kelly, Lorraine Goeuriot, Hanna Suominen, Tobias Schreck, Gondy Leroy, Danielle L Mowery, Sumithra Velupillai, Wendy W Chapman, David Martinez, Guido Zuccon, et al. 2014. Overview of the share/clef ehealth evaluation lab 2014. In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 172– 191.
- Srinivasa Rao Kundeti, J Vijayananda, Srikanth Mujjiga, and M Kalyan. 2016. Clinical named entity recognition: Challenges and opportunities. In Big Data (Big Data), 2016 IEEE International Conference on, pages 1937–1945.
- Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. arXiv preprint arXiv:1603.01360.
- Heeyoung Lee, Yves Peirsman, Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2011. Stanford’s multi-pass sieve coreference resolution system at the conll-2011 shared task. In Proceedings of the fifteenth conference on computational natural language learning: Shared task, pages 28–34.
- Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: pre-trained biomedical language representation model for biomedical text mining. arXiv preprint arXiv:1901.08746.
- Zengjian Liu, Ming Yang, Xiaolong Wang, Qingcai Chen, Buzhou Tang, Zhe Wang, and Hua Xu. 2017. Entity recognition from clinical texts via recurrent neural network. BMC medical informatics and decision making, 17(2):67.
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119.
- Serguei Pakhomov, Bridget McInnes, Terrence Adam, Ying Liu, Ted Pedersen, and Genevieve B Melton. 2010. Semantic similarity and relatedness between clinical terms: an experimental study. In AMIA annual symposium proceedings, volume 2010, page 572. American Medical Informatics Association.
- Serguei VS Pakhomov, Greg Finley, Reed McEwan, Yan Wang, and Genevieve B Melton. 2016. Corpus domain effects on distributional semantic modeling of medical terms. Bioinformatics, 32(23):3635– 3644.
- Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543.
- Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365.
- Sameer Pradhan, Noemie Elhadad, Wendy Chapman, Suresh Manandhar, and Guergana Savova. 2014. Semeval-2014 task 7: Analysis of clinical text. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 54–62.
- Bryan Rink, Sanda Harabagiu, and Kirk Roberts. 2011. Automatic extraction of relations between medical concepts in clinical texts. Journal of the American Medical Informatics Association, 18(5):594–600.
- Kirk Roberts. 2016. Assessing the corpus size vs. similarity trade-off for word embeddings in clinical nlp. In Proceedings of the Clinical Natural Language Processing Workshop (ClinicalNLP), pages 54–63.
- Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149–5152. IEEE.
- Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603.
- Yikang Shen, Wenge Rong, Nan Jiang, Baolin Peng, Jie Tang, and Zhang Xiong. 2017. Word embedding based correlation model for question/answer matching. In AAAI, pages 3511–3517.
- Yuqi Si and Kirk Roberts. 2018. A Frame-Based NLP System for Cancer-Related Information Extraction. In AMIA Annual Symposium Proceedings, volume 2018, page 1524. American Medical Informatics Association.
- Amber Stubbs, Christopher Kotfila, and Ozlem Uzuner. 2015. Automated systems for the de-identification of longitudinal clinical narratives: Overview of 2014 i2b2/uthealth shared task track 1. Journal of biomedical informatics, 58:S11–S19.
- Weiyi Sun, Anna Rumshisky, and Ozlem Uzuner. 2013. Evaluating temporal relations in clinical text: 2012 i2b2 challenge. Journal of the American Medical Informatics Association, 20(5):806–813.
- Hanna Suominen, Sanna Salantera, Sumithra Velupillai, Wendy W Chapman, Guergana Savova, Noemie Elhadad, Sameer Pradhan, Brett R South, Danielle L Mowery, Gareth JF Jones, et al. 2013. Overview of the share/clef ehealth evaluation lab 2013. In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 212– 231.
- Buzhou Tang, Hongxin Cao, Yonghui Wu, Min Jiang, and Hua Xu. 2013. Recognizing clinical entities in hospital discharge summaries using structural support vector machines with word representation features. BMC Medical Informatics and Decision Making, 13(1):S1.
- Buzhou Tang, Qingcai Chen, Xiaolong Wang, Yonghui Wu, Yaoyun Zhang, Min Jiang, Jingqi Wang, and Hua Xu. 2015. Recognizing disjoint clinical concepts in clinical text using machine learning-based methods. In AMIA Annual Symposium Proceedings, volume 2015, page 1184.
- Inigo Jauregi Unanue, Ehsan Zare Borzeshi, and Massimo Piccardi. 2017. Recurrent neural networks with specialized word embeddings for healthdomain named-entity recognition. Journal of biomedical informatics, 76:102–109.
- Ozlem Uzuner, Brett R South, Shuying Shen, and Scott L DuVall. 2011. 2010 i2b2/va challenge on concepts, assertions, and relations in clinical text. Journal of the American Medical Informatics Association, 18(5):552–556.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008.
- Sumithra Velupillai, Hanna Suominen, Maria Liakata, Angus Roberts, Anoop D Shah, Katherine Morley, David Osborn, Joseph Hayes, Robert Stewart, Johnny Downs, et al. 2018. Using clinical natural language processing for health outcomes research: Overview and actionable suggestions for future advances. Journal of biomedical informatics, 88:11– 19.
- Yanshan Wang, Sijia Liu, Naveed Afzal, Majid Rastegar-Mojarad, Liwei Wang, Feichen Shen, Paul Kingsbury, and Hongfang Liu. 2018a. A comparison of word embeddings for the biomedical natural language processing. Journal of biomedical informatics, 87:12–20.
- Yanshan Wang, Liwei Wang, Majid Rastegar-Mojarad, Sungrim Moon, Feichen Shen, Naveed Afzal, Sijia Liu, Yuqun Zeng, Saeed Mehrabi, Sunghwan Sohn, et al. 2018b. Clinical information extraction applications: a literature review. Journal of biomedical informatics, 77:34–49.
- Yonghui Wu, Jun Xu, Min Jiang, Yaoyun Zhang, and Hua Xu. 2015. A study of neural word embeddings for named entity recognition in clinical text. In AMIA Annual Symposium Proceedings, volume 2015, page 1326.
- Hua Xu, Zhenming Fu, Anushi Shah, Yukun Chen, Neeraja B Peterson, Qingxia Chen, Subramani Mani, Mia A Levy, Qi Dai, and Josh C Denny. 2011. Extracting and integrating data from entire electronic health records for detecting colorectal cancer cases. In AMIA Annual Symposium Proceedings, volume 2011, page 1564.
- Yaoyun Zhang, Jingqi Wang, Buzhou Tang, Yonghui Wu, Min Jiang, Yukun Chen, and Hua Xu. 2014. UTH CCB: a report for semeval 2014–task 7 analysis of clinical text. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 802–806.
- Henghui Zhu, Ioannis Ch Paschalidis, and Amir Tahmasebi. 2018. Clinical concept extraction with contextual word embedding. arXiv preprint arXiv:1810.10566.