A Lightweight Neural Model for Biomedical Entity Linking

Cited by: 0|Views11
Weibo:
We propose a simple and lightweight neural model for biomedical entity linking

Abstract:

Biomedical entity linking aims to map biomedical mentions, such as diseases and drugs, to standard entities in a given knowledge base. The specific challenge in this context is that the same biomedical entity can have a wide range of names, including synonyms, morphological variations, and names with different word orderings. Recently, ...More

Code:

Data:

0
Full Text
Bibtex
Weibo
Introduction
  • Entity linking (Entity Normalization) is the task of mapping entity mentions in text documents to standard entities in a given knowledge base.
  • Entity linking maps mentions of diseases, drugs, and measures to normalized entities in standard vocabularies.
  • It is an important ingredient for automation in medical practice, research, and public health.
  • If a medication appears with different names, researchers cannot study its impact, and patients may erroneously be prescribed the same medication twice
Highlights
  • Entity linking (Entity Normalization) is the task of mapping entity mentions in text documents to standard entities in a given knowledge base
  • Given the text “Paris is the son of King Priam”, the goal is to determine that, in this sentence, the word refers to the Greek hero, and to link the word to the corresponding entity in a knowledge base such as YAGO (Suchanek, Kasneci, and Weikum 2007) or DBpedia (Auer et al 2007)
  • We introduce a very lightweight model that achieves a performance statistically indistinguishable from the state-of-the-art Bidirectional Encoder Representations from Transformers (BERT)-based models
  • We propose a simple and lightweight neural model for biomedical entity linking
  • Our experimental results on three standard evaluation benchmarks show that the model is very effective, and achieves a performance that is statistically indistinguishable from the state of the art
  • Future work to improve the architecture can explore 1) automatically assigning a weight for each word in the mentions and entity names to capture the importance of each word, depending, e.g., on its grammatical role; 2) Graph Convolutional Networks (GCNs) (Kipf and Welling 2016; Wu et al 2020) to capture graph structure across mentions and improve our notion of entity coherence
Methods
  • The authors evaluate the model on three datasets.
  • The ShARe/CLEF corpus (Pradhan et al 2013) comprises 199 medical reports for training and 99 for testing.
  • The authors use the July 6, 2012 version of MEDIC (Davis et al 2012), which contains 9,664 disease concepts.
  • The TAC 2017 Adverse Reaction Extraction (ADR) dataset consists of a training set of 101 labels and a test set of 99 labels.
  • The mentions have been mapped manually to the MedDRA 18.1 KB, which contains 23,668 unique concepts
Results
  • 5.1 Overall Performance

    During the candidate generation, the authors generate 20 candidates for each mention.
  • The authors conclude that the candidate generation does not eliminate too many correct candidates.
  • For each dataset, several methods perform within the margin of the best-performing model.
  • Only two models are consistently within the margin across all datasets: BERT and the method.
  • Adding extra features to the base model yields a small increase on the three datasets.
  • Overall, even the base model achieves a performance that is statistically indistinguishable from the state of the art
Conclusion
  • The authors propose a simple and lightweight neural model for biomedical entity linking.
  • The authors' experimental results on three standard evaluation benchmarks show that the model is very effective, and achieves a performance that is statistically indistinguishable from the state of the art.
  • BERT-based models, e.g., have 23 times more parameters and require 6.4 times more computing time for inference.
  • Future work to improve the architecture can explore 1) automatically assigning a weight for each word in the mentions and entity names to capture the importance of each word, depending, e.g., on its grammatical role; 2) Graph Convolutional Networks (GCNs) (Kipf and Welling 2016; Wu et al 2020) to capture graph structure across mentions and improve the notion of entity coherence
Summary
  • Introduction:

    Entity linking (Entity Normalization) is the task of mapping entity mentions in text documents to standard entities in a given knowledge base.
  • Entity linking maps mentions of diseases, drugs, and measures to normalized entities in standard vocabularies.
  • It is an important ingredient for automation in medical practice, research, and public health.
  • If a medication appears with different names, researchers cannot study its impact, and patients may erroneously be prescribed the same medication twice
  • Objectives:

    The authors' goal is to find an optimal θ, which makes the score difference between positive and negative entity candidates as large as possible.
  • Methods:

    The authors evaluate the model on three datasets.
  • The ShARe/CLEF corpus (Pradhan et al 2013) comprises 199 medical reports for training and 99 for testing.
  • The authors use the July 6, 2012 version of MEDIC (Davis et al 2012), which contains 9,664 disease concepts.
  • The TAC 2017 Adverse Reaction Extraction (ADR) dataset consists of a training set of 101 labels and a test set of 99 labels.
  • The mentions have been mapped manually to the MedDRA 18.1 KB, which contains 23,668 unique concepts
  • Results:

    5.1 Overall Performance

    During the candidate generation, the authors generate 20 candidates for each mention.
  • The authors conclude that the candidate generation does not eliminate too many correct candidates.
  • For each dataset, several methods perform within the margin of the best-performing model.
  • Only two models are consistently within the margin across all datasets: BERT and the method.
  • Adding extra features to the base model yields a small increase on the three datasets.
  • Overall, even the base model achieves a performance that is statistically indistinguishable from the state of the art
  • Conclusion:

    The authors propose a simple and lightweight neural model for biomedical entity linking.
  • The authors' experimental results on three standard evaluation benchmarks show that the model is very effective, and achieves a performance that is statistically indistinguishable from the state of the art.
  • BERT-based models, e.g., have 23 times more parameters and require 6.4 times more computing time for inference.
  • Future work to improve the architecture can explore 1) automatically assigning a weight for each word in the mentions and entity names to capture the importance of each word, depending, e.g., on its grammatical role; 2) Graph Convolutional Networks (GCNs) (Kipf and Welling 2016; Wu et al 2020) to capture graph structure across mentions and improve the notion of entity coherence
Tables
  • Table1: Dataset Statistics
  • Table2: Performance of different models. Results in gray are not statistically different from the top result
  • Table3: Ablation study the components of the base model is shown above the gray line; the addition of extra features (Section 3.4) below. If we remove the Alignment Layer (underlined), the accuracy drops the most, with up to 4.06 percentage points. This indicates that the alignment layer can effectively capture the similarity of the corresponding parts of mentions and entity names. The CNN Layer extracts the key components of the names, and removing this part causes a drop of up to 1.87 percentage points. The character-level feature captures morphological variations, and removing it results in a decrease of up to 1.21 percentage points. Therefore, we conclude that all components of our base model are necessary
  • Table4: Performance in the face of typos: Simulated ADR Datasets
  • Table5: Number of model parameters and observed inference time
Download tables as Excel
Related work
  • In the biomedical domain, much early research focuses on capturing string similarity of mentions and entity names with rule-based systems (Dogan and Lu 2012; Kang et al 2013; D’Souza and Ng 2015). Rule-based systems are simple and transparent, but researchers need to define rules manually, and these are bound to an application.

    To avoid manual rules, machine-learning approaches learn suitable similarity measures between mentions and entity names automatically from training sets (Leaman, Islamaj Dogan, and Lu 2013; Dogan, Leaman, and Lu 2014; Ghiasvand and Kate 2014; Leaman and Lu 2016). However, one drawback of these methods is that they cannot recognize semantically related words.

    Recently, deep learning methods have been successfully applied to different NLP tasks, based on pre-trained word embeddings, such as word2vec (Mikolov et al 2013) and Glove (Pennington, Socher, and Manning 2014). Li et al (2017) and Wright (2019) introduce a CNN and RNN, respectively, with pre-trained word embeddings, which casts biomedical entity linking into a ranking problem.
Funding
  • This project was partially funded by the DirtyData project (ANR-17-CE23-0018-01)
Study subjects and analysis
datasets: 3
Yet, we show that our model is competitive with previous work on standard evaluation benchmarks. 4.1 Datasets and Metrics.

We evaluate our model on three datasets (shown in Table 1)
. The ShARe/CLEF corpus (Pradhan et al 2013) comprises 199 medical reports for training and 99 for testing

datasets: 3
As before, we concatenate this score to the vector fout. More precisely, we pre-trained separate entity embeddings for the three datasets and used the mean value of all entity embeddings to represent missing entities. 3.5 NIL Problem

datasets: 3
4.1 Datasets and Metrics. We evaluate our model on three datasets (shown in Table 1). The ShARe/CLEF corpus (Pradhan et al 2013) comprises 199 medical reports for training and 99 for testing

datasets: 3
However, only two models are consistently within the margin across all datasets: BERT and our method. Adding extra features (prior, context, coherence) to our base model yields a small increase on the three datasets. However, overall, even our base model achieves a performance that is statistically indistinguishable from the state of the art

datasets: 3
To understand the effect of each component of our model, we measured the performance of our model when individual components are removed or added. The results of this ablation study on all three datasets are shown in Table 3. The gray row is the accuracy of our base model

data sets: 3
The second column in the table shows the number of parameters of different models. Our model uses an average of only 4.6M parameters across the three data sets, which is 1.6x to 72.9x smaller than the other models. The third column to the tenth column show the practical inference time of the models on the CPU and GPU

datasets: 3
The CPU is described in Section 4.2, and the GPU we used is a single NVIDIA Tesla V100 (32G). Our model is consistently the fastest across all three datasets, both for CPU and GPU (except in the fourth column). On average, our model is 6.4x faster than other BERT models, and our model is much lighter on the CPU

Reference
  • Auer, S.; Bizer, C.; Kobilarov, G.; Lehmann, J.; Cyganiak, R.; and Ives, Z. 2007. Dbpedia: A nucleus for a web of open data. In The semantic web, 722–735. Springer.
    Google ScholarLocate open access versionFindings
  • Bodenreider, O. 2004. The unified medical language system (UMLS): integrating biomedical terminology. Nucleic acids research 32(suppl 1): D267–D270.
    Google ScholarFindings
  • Chen, Q.; Zhu, X.; Ling, Z.; Wei, S.; Jiang, H.; and Inkpen, D. 2016. Enhanced lstm for natural language inference. arXiv preprint arXiv:1609.06038.
    Findings
  • Davis, A. P.; Wiegers, T. C.; Rosenstein, M. C.; and Mattingly, C. J. 2012. MEDIC: a practical disease vocabulary used at the Comparative Toxicogenomics Database. Database 2012.
    Google ScholarLocate open access versionFindings
  • Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
    Findings
  • Dogan, R. I.; Leaman, R.; and Lu, Z. 2014. NCBI disease corpus: a resource for disease name recognition and concept normalization. Journal of biomedical informatics 47: 1–10.
    Google ScholarLocate open access versionFindings
  • Dogan, R. I.; and Lu, Z. 2012. An inference method for disease name normalization. In 2012 AAAI Fall Symposium Series.
    Google ScholarLocate open access versionFindings
  • D’Souza, J.; and Ng, V. 2015. Sieve-based entity linking for the biomedical domain. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), 297–302.
    Google ScholarLocate open access versionFindings
  • Ghiasvand, O.; and Kate, R. J. 2014. R.: UWM: Disorder mention extraction from clinical text using CRFs and normalization using learned edit distance patterns. In In: Proc. SemEval 2014. Citeseer.
    Google ScholarFindings
  • Graves, A.; Mohamed, A.-r.; and Hinton, G. 2013. Speech recognition with deep recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing, 6645–6649. IEEE.
    Google ScholarLocate open access versionFindings
  • He, K.; Zhang, X.; Ren, S.; and Sun, J. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, 1026–1034.
    Google ScholarLocate open access versionFindings
  • Hoffart, J.; Yosef, M. A.; Bordino, I.; Furstenau, H.; Pinkal, M.; Spaniol, M.; Taneva, B.; Thater, S.; and Weikum, G. 2011. Robust disambiguation of named entities in text. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, 782–792.
    Google ScholarLocate open access versionFindings
  • Huang, K.; Altosaar, J.; and Ranganath, R. 2019. Clinicalbert: Modeling clinical notes and predicting hospital readmission. arXiv preprint arXiv:1904.05342.
    Findings
  • Ji, Z.; Wei, Q.; and Xu, H. 2020. Bert-based ranking for biomedical entity normalization. AMIA Summits on Translational Science Proceedings 2020: 269.
    Google ScholarLocate open access versionFindings
  • Jiao, X.; Yin, Y.; Shang, L.; Jiang, X.; Chen, X.; Li, L.; Wang, F.; and Liu, Q. 2019. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.10351.
    Findings
  • Kang, N.; Singh, B.; Afzal, Z.; van Mulligen, E. M.; and Kors, J. A. 2013. Using rule-based natural language processing to improve disease normalization in biomedical text. Journal of the American Medical Informatics Association 20(5): 876–881.
    Google ScholarLocate open access versionFindings
  • Kim, Y. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882.
    Findings
  • Kipf, T. N.; and Welling, M. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.
    Findings
  • Lan, Z.; Chen, M.; Goodman, S.; Gimpel, K.; Sharma, P.; and Soricut, R. 20Albert: A lite bert for selfsupervised learning of language representations. arXiv preprint arXiv:1909.11942.
    Findings
  • Leaman, R.; Islamaj Dogan, R.; and Lu, Z. 2013. DNorm: disease name normalization with pairwise learning to rank. Bioinformatics 29(22): 2909–2917.
    Google ScholarLocate open access versionFindings
  • Leaman, R.; and Lu, Z. 2016. TaggerOne: joint named entity recognition and normalization with semi-Markov Models. Bioinformatics 32(18): 2839–2846.
    Google ScholarLocate open access versionFindings
  • Lee, J.; Yoon, W.; Kim, S.; Kim, D.; Kim, S.; So, C. H.; and Kang, J. 2020. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36(4): 1234–1240.
    Google ScholarLocate open access versionFindings
  • Li, H.; Chen, Q.; Tang, B.; Wang, X.; Xu, H.; Wang, B.; and Huang, D. 2017. CNN-based ranking for biomedical entity normalization. BMC bioinformatics 18(11): 79–86.
    Google ScholarLocate open access versionFindings
  • Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G. S.; and Dean, J. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, 3111–3119.
    Google ScholarLocate open access versionFindings
  • Pennington, J.; Socher, R.; and Manning, C. D. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 1532–1543.
    Google ScholarLocate open access versionFindings
  • Pradhan, S.; Elhadad, N.; South, B. R.; Martinez, D.; Christensen, L. M.; Vogel, A.; Suominen, H.; Chapman, W. W.; and Savova, G. K. 2013. Task 1: ShARe/CLEF eHealth Evaluation Lab 2013. In CLEF (Working Notes), 212–31.
    Google ScholarFindings
  • Sohn, S.; Comeau, D. C.; Kim, W.; and Wilbur, W. J. 2008. Abbreviation definition identification based on automatic precision estimates. BMC bioinformatics 9(1): 402.
    Google ScholarLocate open access versionFindings
  • Suchanek, F. M.; Kasneci, G.; and Weikum, G. 2007. Yago: a core of semantic knowledge. In Proceedings of the 16th international conference on World Wide Web, 697–706.
    Google ScholarLocate open access versionFindings
  • Sun, Z.; Yu, H.; Song, X.; Liu, R.; Yang, Y.; and Zhou, D. 2020. Mobilebert: a compact task-agnostic bert for resourcelimited devices. arXiv preprint arXiv:2004.02984.
    Findings
  • Tai, K. S.; Socher, R.; and Manning, C. D. 2015. Improved semantic representations from tree-structured long shortterm memory networks. arXiv preprint arXiv:1503.00075.
    Findings
  • Wang, S.; and Jiang, J. 2016. A compare-aggregate model for matching text sequences. arXiv preprint arXiv:1611.01747.
    Findings
  • Wright, D. 2019. NormCo: Deep disease normalization for biomedical knowledge base construction. Ph.D. thesis, UC San Diego.
    Google ScholarFindings
  • Wu, J.; Zhang, R.; Mao, Y.; Guo, H.; Soflaei, M.; and Huai, J. 2020. Dynamic Graph Convolutional Networks for Entity Linking. In Proceedings of The Web Conference 2020, 1149–1159.
    Google ScholarLocate open access versionFindings
  • Xu, J.; Lee, H.-J.; Ji, Z.; Wang, J.; Wei, Q.; and Xu, H. 2017. UTH CCB System for Adverse Drug Reaction Extraction from Drug Labels at TAC-ADR 2017. In TAC.
    Google ScholarFindings
  • Zhang, Y.; Chen, Q.; Yang, Z.; Lin, H.; and Lu, Z. 2019. BioWordVec, improving biomedical word embeddings with subword information and MeSH. Scientific data 6(1): 1–9.
    Google ScholarLocate open access versionFindings
Your rating :
0

 

Tags
Comments