Using Clinical Natural Language Processing for Health Outcomes Research: Overview and Actionable Suggestions for Future Advances

Journal of Biomedical Informatics, pp. 11-19, 2018.

Cited by: 27|Bibtex|Views31|DOI:https://doi.org/10.1016/j.jbi.2018.10.005
WOS EI
Other Links: pubmed.ncbi.nlm.nih.gov|dblp.uni-trier.de|academic.microsoft.com|www.sciencedirect.com
Weibo:
For clinical Natural Language Processing method development to become more integral in clinical outcomes research, there is a need to develop evaluation workbenches that can be used by clinicians to better understand the underlying parts of an Natural Language Processing system a...

Abstract:

•Natural Language Processing methods can be impactful in health outcomes research.•More rigorous evaluation practices are necessary to advance this field.•Synthetic data and novel governance structures could address data access challenges.•Evaluation workbenches that allow for extrinsic and detailed evaluation are needed.•Structured proto...More

Code:

Data:

0
Introduction
  • Appropriate utilization of large data sources such as Electronic

    Health Record databases could have a dramatic impact on health care research and delivery.
  • The above include recommendations to address the key challenges of limited collaboration, lack of shared resources and evaluation-approaches of crucial tasks, such as de-identification, recognition and classification of medical concepts, semantic modifiers, and temporal information.
  • These challenges have been addressed by the organization of several shared tasks
  • These include the Informatics for Integrating Biology and the Bedside (i2b2) challenges [59], the Conference and Labs of the Evaluation Forum (CLEF) eHealth challenges [1013], and the Semantic Evaluation (SemEval) challenges [1416].
  • These efforts have enabled a valuable platform for international NLP method development
Highlights
  • Appropriate utilization of large data sources such as Electronic

    Health Record databases could have a dramatic impact on health care research and delivery
  • Studies of diagnostic tools are most similar to Natural Language Processing method development - testing whether a history item, examination finding or test result is associated with a subsequent diagnosis
  • For clinical Natural Language Processing method development to become more integral in clinical outcomes research, there is a need to develop evaluation workbenches that can be used by clinicians to better understand the underlying parts of an Natural Language Processing system and its impact on outcomes
  • Work in the general Natural Language Processing domain could be inspirational for such development, for instance integrating methods to analyse the effect of Natural Language Processing pipeline steps in downstream tasks such as the effect of dependency parsing approaches [84]
  • Facilitating the integration of domain knowledge in Natural Language Processing system development can be done by providing support for formalized knowledge representations that can be used in subsequent Natural Language Processing method development [86]
  • We propose a minimal structured protocol that could be used when reporting clinical Natural Language Processing method development and its evaluation, to enable transparency and reproducibility
Methods
  • Methods for developing shareable data

    Risks for compromised privacy are evident in analyzing text from health records and in big data health research more generally.
  • Methods for developing shareable data.
  • Risks for compromised privacy are evident in analyzing text from health records and in big data health research more generally.
  • Synthetic data has been successful in tasks such as dialogue generation [59] and is a promising direction at least as a complement for method development where access to data is challenging
Results
  • Evaluation paradigms

    All empirical research studies need to be evaluated in order to allow for scientific assessment of a study.
  • The most basic underlying construction for quantitative validation in both fields is a 2 × 2 contingency table, where the number of correctly and incorrectly assigned values for a given binary outcome or classification label is compared with a gold standard, i.e. the set of ’true’ or correct values
  • This table can be used to calculate performance metrics such as precision (Positive Predictive Value), recall, accuracy, F-score, and specificity.
  • Facilitating the integration of domain knowledge in NLP system development can be done by providing support for formalized knowledge representations that can be used in subsequent NLP method development [86]
Conclusion
  • The authors have sought to provide a broad outline of the current state-ofthe-art, opportunities, challenges, and needs in the use of NLP for health outcomes research, with a particular focus on evaluation methods.
  • The authors have outlined methodological aspects from a clinical as well as an NLP perspective and identify three main challenges: data availability, evaluation workbenches and reporting standards.
  • The authors propose a minimal structured protocol that could be used when reporting clinical NLP method development and its evaluation, to enable transparency and reproducibility.
  • The authors envision further advances in methods for data access, evaluation methods that move beyond current intrinsic metrics and move closer to clinical practice and utility, and in transparent and reproducible method development
Summary
  • Introduction:

    Appropriate utilization of large data sources such as Electronic

    Health Record databases could have a dramatic impact on health care research and delivery.
  • The above include recommendations to address the key challenges of limited collaboration, lack of shared resources and evaluation-approaches of crucial tasks, such as de-identification, recognition and classification of medical concepts, semantic modifiers, and temporal information.
  • These challenges have been addressed by the organization of several shared tasks
  • These include the Informatics for Integrating Biology and the Bedside (i2b2) challenges [59], the Conference and Labs of the Evaluation Forum (CLEF) eHealth challenges [1013], and the Semantic Evaluation (SemEval) challenges [1416].
  • These efforts have enabled a valuable platform for international NLP method development
  • Methods:

    Methods for developing shareable data

    Risks for compromised privacy are evident in analyzing text from health records and in big data health research more generally.
  • Methods for developing shareable data.
  • Risks for compromised privacy are evident in analyzing text from health records and in big data health research more generally.
  • Synthetic data has been successful in tasks such as dialogue generation [59] and is a promising direction at least as a complement for method development where access to data is challenging
  • Results:

    Evaluation paradigms

    All empirical research studies need to be evaluated in order to allow for scientific assessment of a study.
  • The most basic underlying construction for quantitative validation in both fields is a 2 × 2 contingency table, where the number of correctly and incorrectly assigned values for a given binary outcome or classification label is compared with a gold standard, i.e. the set of ’true’ or correct values
  • This table can be used to calculate performance metrics such as precision (Positive Predictive Value), recall, accuracy, F-score, and specificity.
  • Facilitating the integration of domain knowledge in NLP system development can be done by providing support for formalized knowledge representations that can be used in subsequent NLP method development [86]
  • Conclusion:

    The authors have sought to provide a broad outline of the current state-ofthe-art, opportunities, challenges, and needs in the use of NLP for health outcomes research, with a particular focus on evaluation methods.
  • The authors have outlined methodological aspects from a clinical as well as an NLP perspective and identify three main challenges: data availability, evaluation workbenches and reporting standards.
  • The authors propose a minimal structured protocol that could be used when reporting clinical NLP method development and its evaluation, to enable transparency and reproducibility.
  • The authors envision further advances in methods for data access, evaluation methods that move beyond current intrinsic metrics and move closer to clinical practice and utility, and in transparent and reproducible method development
Funding
  • This work is the result of an international workshop held at the Institute of Psychiatry, Psychology and Neuroscience, King’s College London, on April 28th 2017, with financial support from the European Science Foundation (ESF) Research Networking Programme Evaluating Information Access Systems: http://eliasnetwork.eu/
  • SV is supported by the Swedish Research Council (2015-00359) and the Marie Skłodowska Curie Actions, Cofund, Project INCA 600398
  • RD is funded by a Clinician Scientist Fellowship (research project e-HOST-IT) from the Health Foundation in partnership with the Academy of Medical Sciences
  • JD is supported by a Medical Research Council (MRC) Clinical Research Training Fellowship (MR/L017105/1)
  • KM is funded by the Wellcome Trust Seed Award in Science [109823/Z/15/Z]
  • RS and AR is part-funded by the National Institute for Health Research (NIHR) Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and King’s College London
  • David (PJ) Osborn, Joseph (F) Hayes and Anoop (D) Shah are supported by the National Institute for Health Research University College London Hospitals Biomedical Research Centre
  • Prof Osborn is also in part supported by the NIHR Collaboration for Leadership in Applied Health Research and Care (CLAHRC) North Thames at Bart’s Health NHS Trust
Reference
  • A. Névéol, P. Zweigenbaum, Clinical Natural Language Processing in 2014: foundational methods supporting efficient healthcare, Yearb. Med. Inform. 10 (1) (2015) 194–198.
    Google ScholarLocate open access versionFindings
  • S. Velupillai, D. Mowery, B.R. South, M. Kvist, H. Dalianis, Recent advances in clinical natural language processing in support of semantic analysis, IMIA Yearb. Med. Inform. 10 (2015) 183–193.
    Google ScholarLocate open access versionFindings
  • W.W. Chapman, P.M. Nadkarni, L. Hirschman, L.W. D’Avolio, G.K. Savova, O. Uzuner, Overcoming barriers to NLP for clinical text: the role of shared tasks and the need for additional creative solutions, J. Am. Med. Inform. Assoc. 18 (5) (2011) 540–543.
    Google ScholarLocate open access versionFindings
  • C. Friedman, T.C. Rindflesch, M. Corn, Natural language processing: State of the art and prospects for significant progress, a workshop sponsored by the National Library of Medicine, J. Biomed. Inform. 46 (5) (2013) 765–773, https://doi.org/10.1016/j.jbi.2013.06.004.
    Locate open access versionFindings
  • Ö. Uzuner, Y. Luo, P. Szolovits, Evaluating the state-of-the-art in automatic deidentification, J. Am. Med. Inform. Assoc. 14 (5) (2007) 550, https://doi.org/10.1197/jamia.M2444.
    Locate open access versionFindings
  • Ö. Uzuner, I. Solti, E. Cadag, Extracting medication information from clinical text, J. Am. Med. Inform. Assoc. 17 (5) (2010) 514, https://doi.org/10.1136/jamia.2010.003947.
    Locate open access versionFindings
  • Ö. Uzuner, B.R. South, S. Shen, S.L. DuVall, 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text, J. Am. Med. Inform. Assoc. 18 (5) (2011) 552, https://doi.org/10.1136/amiajnl-2011-000203.
    Locate open access versionFindings
  • Ö. Uzuner, A. Bodnari, S. Shen, T. Forbush, J. Pestian, B.R. South, Evaluating the state of the art in coreference resolution for electronic medical records, J. Am. Med. Inform. Assoc. 19 (5) (2012) 786, https://doi.org/10.1136/amiajnl-2011-000784.
    Locate open access versionFindings
  • W. Sun, A. Rumshisky, O. Uzuner, Evaluating temporal relations in clinical text: 2012 i2b2 Challenge, J. Am. Med. Inform. Assoc. 20 (5) (2013) 806, https://doi.org/10.1136/amiajnl-2013-001628.
    Locate open access versionFindings
  • H. Suominen, S. Salanterä, S. Velupillai, W. Chapman, G. Savova, N. Elhadad, S. Pradhan, B. South, D. Mowery, G. Jones, J. Leveling, L. Kelly, L. Goeuriot, D. Martinez, G. Zuccon, Overview of the ShARe/CLEF eHealth evaluation lab 2013, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 8138 LNCS (2013) 212–231. https://doi.org/10.1007/978-3-642-40802-1_24.
    Locate open access versionFindings
  • L. Kelly, L. Goeuriot, H. Suominen, T. Schreck, G. Leroy, D. Mowery, S. Velupillai, W. Chapman, D. Martinez, G. Zuccon, J. Palotti, Overview of the ShARe/CLEF eHealth evaluation lab 2014, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 8685 LNCS (2014) 172–191. https://doi.org/10.1007/978-3-319-11382-1_17.
    Locate open access versionFindings
  • L. Goeuriot, L. Kelly, H. Suominen, L. Hanlen, A. Névéol, C. Grouin, J. Palotti, G. Zuccon, Overview of the CLEF eHealth Evaluation Lab 2015, Springer International Publishing, 2015 pp. 429–443.
    Google ScholarFindings
  • L. Kelly, L. Goeuriot, H. Suominen, A. Névéol, J. Palotti, G. Zuccon, Overview of the CLEF eHealth Evaluation Lab 2016, Springer International Publishing, 2016, https://doi.org/10.1007/978-3-319-44564-9_24 pp.255–266.
    Findings
  • N. Elhadad, S. Pradhan, S. Gorman, S. Manandhar, W. Chapman, G. Savova, SemEval-2015 task 14: Analysis of clinical text, Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), Association for Computational Linguistics, Denver, Colorado, 2015, pp. 303–310.
    Google ScholarLocate open access versionFindings
  • S. Bethard, L. Derczynski, G. Savova, J. Pustejovsky, M. Verhagen, SemEval-2015 task 6: Clinical TempEval, Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), Association for Computational Linguistics, Denver, Colorado, 2015, pp. 806–814.
    Google ScholarLocate open access versionFindings
  • S. Bethard, G. Savova, W.-T. Chen, L. Derczynski, J. Pustejovsky, M. Verhagen, Semeval-2016 task 12: Clinical tempeval, Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), Association for Computational Linguistics, San Diego, California, 2016, pp. 1052–1062.
    Google ScholarLocate open access versionFindings
  • H. Wu, G. Toti, K.I. Morley, Z.M. Ibrahim, A. Folarin, R. Jackson, I. Kartoglu, A. Agrawal, C. Stringer, D. Gale, G. Gorrell, A. Roberts, M. Broadbent, R. Stewart, R.J. Dobson, SemEHR: a general-purpose semantic search system to surface semantic data from clinical notes for tailored care, trial recruitment, and clinical research, J. Am. Med. Inform. Assoc. 25 (5) (2018) 530–537, https://doi.org/10.1093/jamia/ocx160 arXiv:/oup/backfile/content_public/journal/jamia/25/5/10.1093_jamia_ocx160/3/ocx160.pdf.
    Locate open access versionFindings
  • M. De Choudhury, S. De, Mental health discourse on reddit: self-disclosure, social support, and anonymity, in: Eighth International AAAI Conference on Weblogs and Social Media, 2014.
    Google ScholarLocate open access versionFindings
  • U. Pavalanathan, M. De Choudhury, Identity Management and Mental Health Discourse in Social Media, in: Proceedings of the International World-Wide Web Conference. International WWW Conference 2015 (Companion), 2015, pp. 315–321.
    Google ScholarLocate open access versionFindings
  • D. Mowery, H. Smith, T. Cheney, G. Stoddard, G. Coppersmith, C. Bryan, M. Conway, Understanding depressive symptoms and psychosocial stressors on twitter: a corpus-based study, J. Med. Internet Res. 19 (2) (2017) e48, https://doi.org/10.2196/jmir.6895.
    Locate open access versionFindings
  • G. Gkotsis, A. Oellrich, S. Velupillai, M. Liakata, T.J.P. Hubbard, R.J.B. Dobson, R. Dutta, Characterisation of mental health conditions in social media using Informed Deep Learning, Sci. Rep. 7 (2017) 45141.
    Google ScholarLocate open access versionFindings
  • C. Howes, M. Purver, R. McCabe, Linguistic Indicators of Severity and Progress in Online Text-based Therapy for Depression, Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, Association for Computational Linguistics, Baltimore, Maryland, USA, 2014, pp. 7–16.
    Google ScholarLocate open access versionFindings
  • D. Angus, B. Watson, A. Smith, C. Gallois, J. Wiles, Visualising conversation structure across time: insights into effective doctor-patient consultations, PLOS One 7 (6) (2012) 1–12, https://doi.org/10.1371/journal.pone.0038014.
    Locate open access versionFindings
  • T. Althoff, K. Clark, J. Leskovec, Natural Language Processing for Mental Health: Large Scale Discourse Analysis of Counseling Conversations, CoRR abs/1605. 04462. URL <http://arxiv.org/abs/1605.04462>.
    Findings
  • E. Yelland, What text mining analysis of psychotherapy records can tell us about therapy process and outcome, Ph.D. thesis, UCL (University College London), 2017.
    Google ScholarFindings
  • J.P. Pestian, P. Matykiewicz, M. Linn-Gust, B. South, O. Uzuner, J. Wiebe, K.B. Cohen, J. Hurdle, C. Brew, Sentiment analysis of suicide notes: a shared task, Biomedical Informatics Insights 5 (2012) 3–16, https://doi.org/10.4137/BII.S9042.
    Locate open access versionFindings
  • D.N. Milne, G. Pink, B. Hachey, R.A. Calvo, CLPsych 2016 shared task: triaging content in online peer-support forums, Proceedings of the Third Workshop on Computational Linguistics and Clinical Psychology, Association for Computational Linguistics, San Diego, CA, USA, 2016, pp. 118–127.
    Google ScholarLocate open access versionFindings
  • M. Filannino, A. Stubbs, Ö. Uzuner, Symptom severity prediction from neuropsychiatric clinical records: Overview of 2016 {CEGS} N-GRID shared tasks Track 2, J. Biomed. Inform. (2017), https://doi.org/10.1016/j.jbi.2017.04.017.
    Locate open access versionFindings
  • H. Suominen, S. Pyysalo, M. Hiissa, F. Ginter, S. Liu, D. Marghescu, T. Pahikkala, B. Back, H. Karsten, T. Salakoski, Performance evaluation measures for text mining, in: M. Song, Y.-F. Wu (Eds.), Handbook of Research on Text and Web Mining Technologies, vol. II, IGI Global, Hershey, Pennsylvania, USA, 2008, pp. 724–747 (Chapter XLI).
    Google ScholarLocate open access versionFindings
  • E. Steyerberg, Clinical Prediction Models, Springer-Verlag, New York, 2009.
    Google ScholarFindings
  • G. Collins, J. Reitsma, D. Altman, K. Moons, Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (tripod): The tripod statement, Ann. Intern. Med. 162 (1) (2015) 55–63, https://doi.org/10.7326/M140697 arXiv:/data/journals/aim/931895/0000605-201501060-00009.pdf, URL +https://doi.org/10.7326/M14-0697.
    Locate open access versionFindings
  • K. Sparck Jones, Evaluating Natural Language Processing Systems An Analysis and Review, Lecture Notes in Computer Science, Lecture Notes in Artificial Intelligence, 1083, 1995.
    Google ScholarLocate open access versionFindings
  • P. Paroubek, S. Chaudiron, L. Hirschman, Editorial: Principles of Evaluation in Natural Language Processing, TAL 48 (1) (2007) 7–31 <http://www.atala.org/ Editorial>.
    Locate open access versionFindings
  • L. Dybkjaer, Evaluation of Text and Speech Systems, Text, Speech and Language Technology, 37, 2007.
    Google ScholarLocate open access versionFindings
  • P.R. Cohen, A.E. Howe, Toward AI research methodology: three case studies in evaluation, IEEE Trans. Syst., Man Cybern. 19 (3) (1989) 634–646.
    Google ScholarFindings
  • G. Hripcsak, A.S. Rothschild, Agreement, the f-measure, and reliability in information retrieval, J. Am. Med. Inform. Assoc. 12 (3) (2005) 296–298.
    Google ScholarLocate open access versionFindings
  • B.A. Goldstein, A.M. Navar, M.J. Pencina, J.P.A. Ioannidis, Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review, J. Am. Med. Inform Assoc.: JAMIA 24 (1) (2017) 198–208, https://doi.org/10.1093/jamia/ocw042.
    Locate open access versionFindings
  • A. Roberts, Language, structure, and reuse in the electronic health record, AMA J. Ethics 19 (3) (2017) 281–288.
    Google ScholarLocate open access versionFindings
  • S.T. Rosenbloom, J.C. Denny, H. Xu, N. Lorenzi, W.W. Stead, K.B. Johnson, Data from clinical notes: a perspective on the tension between structure and flexible documentation, J. Am. Med. Inform. Assoc. 18 (2) (2011) 181–186.
    Google ScholarLocate open access versionFindings
  • T. Greenhalgh, H.W.W. Potts, G. Wong, P. Bark, D. Swinglehurst, Tensions and paradoxes in electronic patient record research: a systematic literature review using the meta-narrative method, Milbank Q 87 (4) (2009) 729–788.
    Google ScholarLocate open access versionFindings
  • G. Carter, A. Milner, K. McGill, J. Pirkis, N. Kapur, M.J. Spittal, Predicting suicidal behaviours using clinical instruments: systematic review and meta-analysis of positive predictive values for risk scales, Br. J. Psychiatry 210 (6) (2017) 387–395, https://doi.org/10.1192/bjp.bp.116.182717 <http://bjp.rcpsych.org/content/210/6/387>.
    Locate open access versionFindings
  • J. Downs, S. Velupillai, G. Gkotsis, R. Holden, M. Kikoler, H. Dean, A. Fernandes, R. Dutta, Detection of suicidality in adolescents with autism spectrum disorders: developing a natural language processing approach for use in electronic health records, AMIA annual Symposium, AMIA, 2017, pp. 641–649.
    Google ScholarFindings
  • S. Gange, E.T. Golub, From smallpox to big data: the next 100 years of epidemiologic methods, Am. J. Epidemiol. 183 (5) (2016) 423–426, https://doi.org/10.1093/aje/kwv150.
    Locate open access versionFindings
  • S.M. Lynch, J.H. Moore, A call for biological data mining approaches in epidemiology, BioData Mining 9 (1) (2016) 1, https://doi.org/10.1186/s13040-0150079-8 <http://www.biodatamining.org/content/9/1/1>.
    Locate open access versionFindings
  • J. Bell, C. Kilic, R. Prabakaran, Y.Y. Wang, R. Wilson, M. Broadbent, A. Kumar, V. Curtis, Use of electronic health records in identifying drug and alcohol misuse among psychiatric in-patients, The Psychiatrist 37 (1) (2013) 15–20, https://doi.org/10.1192/pb.bp.111.038240.
    Locate open access versionFindings
  • E. Ford, J.A. Carroll, H.E. Smith, D. Scott, J.A. Cassell, Extracting information from the text of electronic medical records to improve case detection: a systematic review, J. Am. Med. Inform. Assoc. 23 (5) (2016) 1007–1015, https://doi.org/10.1093/jamia/ocv180.
    Locate open access versionFindings
  • K.P. Liao, T. Cai, G.K. Savova, S.N. Murphy, E.W. Karlson, A.N. Ananthakrishnan, V.S. Gainer, S.Y. Shaw, Z. Xia, P. Szolovits, S. Churchill, I. Kohane, Development of phenotype algorithms using electronic medical records and incorporating natural language processing, BMJ 350 (2015) h1885, https://doi.org/10.1136/bmj.h1885.
    Locate open access versionFindings
  • G. Hripcsak, D.J. Albers, Next-generation phenotyping of electronic health records, J. Am. Med. Inform. Assoc. 20 (1) (2013) 117–121, https://doi.org/10.1136/amiajnl-2012-001145.
    Locate open access versionFindings
  • K.M. Newton, P.L. Peissig, A.N. Kho, S.J. Bielinski, R.L. Berg, V. Choudhary, M. Basford, C.G. Chute, I.J. Kullo, R. Li, J.a. Pacheco, L.V. Rasmussen, L. Spangler, J.C. Denny, Validation of electronic medical record-based phenotyping algorithms: results and lessons learned from the eMERGE network, J. Am. Med. Inform. Assoc. 20 (e1) (2013) e147–54, https://doi.org/10.1136/amiajnl-2012-000896.
    Locate open access versionFindings
  • K.I. Morley, J. Wallace, S.C. Denaxas, R.J. Hunter, R.S. Patel, P. Perel, A.D. Shah, A.D. Timmis, R.J. Schilling, H. Hemingway, Defining disease phenotypes using national linked electronic health records: a case study of atrial fibrillation, PLoS One 9 (11) (2014) e110900, https://doi.org/10.1371/journal.pone.0110900.
    Locate open access versionFindings
  • G. Peat, R.D. Riley, P. Croft, K.I. Morley, P.a. Kyzas, K.G.M. Moons, P. Perel, E.W. Steyerberg, S. Schroter, D.G. Altman, H. Hemingway, Improving the transparency of prognosis research: the role of reporting, data sharing, registration, and protocols, PLoS Med. 11 (7) (2014) e1001671, https://doi.org/10.1371/journal.pmed.1001671.
    Locate open access versionFindings
  • S. Wu, T. Miller, J. Masanz, M. Coarr, S. Halgrim, D. Carrell, C. Clark, Negation’s not solved: generalizability versus optimizability in clinical natural language processing, PLOS One 9 (11) (2014) 1–11, https://doi.org/10.1371/journal.pone.0112774.
    Locate open access versionFindings
  • D. Demner-Fushman, W.W. Chapman, C.J. McDonald, What can natural language processing do for clinical decision support? J. Biomed. Inform. 42 (5) (2009) 760–772, https://doi.org/10.1016/j.jbi.2009.08.007.
    Locate open access versionFindings
  • K. Zheng, V.G.V. Vydiswaran, Y. Liu, Y. Wang, A. Stubbs, O. Uzuner, A.E. Gururaj, S. Bayer, J. Aberdeen, A. Rumshisky, S. Pakhomov, H. Liu, H. Xu, Ease of adoption of clinical natural language processing software: an evaluation of five systems, J. Biomed. Inform. 58 Suppl (2015) S189–96.
    Google ScholarLocate open access versionFindings
  • D.R. Kaufman, B. Sheehan, P. Stetson, A.R. Bhatt, A.I. Field, C. Patel, J.M. Maisel, Natural language processing-enabled and conventional data capture methods for input to electronic health records: a comparative usability study, JMIR Med. Inform. 4 (4) (2016) e35.
    Google ScholarLocate open access versionFindings
  • H. Suominen, H. Müller, L. Ohno-Machado, S. Salanterä, G. Schreier, L. Hanlen, Prerequisites for International Exchanges of Health Information: Comparison of Australian, Austrian, Finnish, Swiss, and US Privacy Policies, in: TBA (Ed.), Medinfo
    Google ScholarFindings
  • H. Suominen, L. Zhou, L. Hanlen, G. Ferraro, Benchmarking clinical speech recognition and information extraction: new data, methods, and evaluations, JMIR Med. Inform. 3 (2) (2015) e19, https://doi.org/10.2196/medinform.4321.
    Locate open access versionFindings
  • E. Aramaki, M. Morita, Y. Kano, T. Ohkuma, Overview of the NTCIR-11 MedNLP task, in: Proceedings of the 11th NTCIR Conference, NII Testbeds and Community for Information access Research (NTCIR), Tokyo, Japan, 2014, pp. 147–154.
    Google ScholarLocate open access versionFindings
  • I. Serban, A. Sordoni, R. Lowe, L. Charlin, J. Pineau, A. Courville, Y. Bengio, A hierarchical latent variable encoder-decoder model for generating dialogues, in: AAAI Conference on Artificial Intelligence, 2017. URL <https://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14567>.
    Locate open access versionFindings
  • O. Uzuner, I. Goldstein, Y. Luo, I. Kohane, Identifying patient smoking status from medical discharge records, J. Am. Med. Inform. Assoc. 15 (1) (2008) 14–24.
    Google ScholarLocate open access versionFindings
  • I. McCowan, D. Moore, M.-J. Fry, Classification of cancer stage from free-text histology reports, Conf. Proc. IEEE Eng. Med. Biol. Soc. 1 (2006) 5153–5156.
    Google ScholarLocate open access versionFindings
  • G. Gkotsis, S. Velupillai, A. Oellrich, H. Dean, M. Liakata, R. Dutta, Don’t let notes be misunderstood: a negation detection method for assessing risk of suicide in mental health records, Proceedings of the Third Workshop on Computational Linguistics and Clinical Psychology, Association for Computational Linguistics, San Diego, CA, USA, 2016, pp. 95–105 <http://www.aclweb.org/anthology/W160310>.
    Locate open access versionFindings
  • H. Kaur, S. Sohn, C.-I. Wi, E. Ryu, M.A. Park, K. Bachman, H. Kita, I. Croghan, J.A. Castro-Rodriguez, G.A. Voge, H. Liu, Y.J. Juhn, Automated chart review utilizing natural language processing algorithm for asthma predictive index, BMC Pulmonary Med. 18 (1) (2018) 34, https://doi.org/10.1186/s12890-018-0593-9.
    Locate open access versionFindings
  • R. Miotto, L. Li, Brian A. Kidd, Joel T. Dudley, Deep patient: an unsupervised representation to predict the future of patients from the electronic health records, Sci. Rep. 6 (2045-2322) (2016), https://doi.org/10.1038/srep26094.
    Locate open access versionFindings
  • B. Elvevag, P.W. Foltz, M. Rosenstein, L.E. Delisi, An automated method to analyze language use in patients with schizophrenia and their first-degree relatives, J. Neurolinguist. 23 (3) (2010) 270–284, https://doi.org/10.1016/j.jneuroling.2009.05.002.
    Locate open access versionFindings
  • C.M. Corcoran, F. Carrillo, D. Fernández-Slezak, G. Bedi, C. Klim, D.C. Javitt, C.E. Bearden, G.A. Cecchi, Prediction of psychosis across protocols and risk cohorts using automated language analysis, World Psychiatry 17 (1) (2018) 67–75, https://doi.org/10.1002/wps.20491 <http://www.ncbi.nlm.nih.gov/pmc/articles/ PMC5775133/>.[67] K.C. Fraser, J.A. Meltzer, N.L. Graham, C. Leonard, G. Hirst, S.E. Black, E. Rochon, Automated classification of primary progressive aphasia subtypes from narrative speech transcripts, Language, Comput. Cognit. Neurosci.55 (2014)43–60, https://doi.org/10.1016/j.cortex.2012.12.006 <http://www.sciencedirect.com/science/article/pii/S0010945212003413>.
    Locate open access versionFindings
  • [68] E. Keuleers, D.A. Balota, Megastudies, crowdsourcing, and large datasets in psycholinguistics: an overview of recent developments, Quart. J. Exp. Psychol. 68 (8) (2015) 1457–1468, https://doi.org/10.1080/17470218.2015.1051065 pMID:25975773.
    Locate open access versionFindings
  • [69] G. Coppersmith, M. Dredze, C. Harman, K. Hollingshead, M. Mitchell, CLPsych 2015 Shared Task: Depression and PTSD on Twitter, Proceedings of the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, Association for Computational Linguistics, Denver, Colorado, 2015, pp. 31–39.
    Google ScholarLocate open access versionFindings
  • [70] A. Benton, M. Mitchell, D. Hovy, Multitask learning for mental health conditions with limited social media data, Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, Long Papers, vol. 1, Association for Computational Linguistics, Valencia, Spain, 2017, pp. 152–162.
    Google ScholarLocate open access versionFindings
  • [71] A. Tsakalidis, M. Liakata, T. Damoulas, B. Jellinek, W. Guo, A. Cristea, Combining Heterogeneous User Generated Data to Sense Well-being, in: Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, The COLING 2016 Organizing Committee, Osaka, Japan, 2016, pp. 3007–3018.
    Google ScholarLocate open access versionFindings
  • [72] L. Canzian, M. Musolesi, Trajectories of depression: unobtrusive monitoring of depressive states by means of smartphone mobility traces analysis, Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp ’15, ACM, New York, NY, USA, 2015, pp. 1293–1304,, https://doi.org/10.1145/2750858.2805845.
    Locate open access versionFindings
  • [73] N. Jaques, S. Taylor, A. Sano, R. Picard, Multi-task, multi-kernel learning for estimating individual wellbeing, in: Proceedings of NIPS Workshop on Multimodal Machine Learning, 2015.
    Google ScholarLocate open access versionFindings
  • [74] N. Jaques, O. Rudovic, S. Taylor, A. Sano, R. Picard, Predicting tomorrow’s mood, health, and stress level using personalized multitask learning and domain adaptation, in: Proc. IJCAI, 2017.
    Google ScholarFindings
  • [75] A. Tsakalidis, M. Liakata, T. Damoulas, A. Cristea, Can we assess mental health through social media and smart devices? Addressing bias in methodology and evaluation, in: Proceedings of ECML-PKDD 2018, the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, The ECML-PKDD Organizing Committee, Dublin, Ireland, 2018.
    Google ScholarLocate open access versionFindings
  • [76] A.E. Johnson, T.J. Pollard, L. Shen, L.-w. H. Lehman, M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. Anthony Celi, R.G. Mark, MIMIC-III, a freely accessible critical care database, Sci. Data 3 (2016) 160035.
    Google ScholarLocate open access versionFindings
  • [77] C.A. Harle, E.H. Golembiewski, K.P. Rahmanian, J.L. Krieger, D. Hagmajer, A.G. Mainous 3rd, R.E. Moseley, Patient preferences toward an interactive e-consent application for research using electronic health records, J. Am. Med. Inform. Assoc. 25 (3) (2018) 360–368, https://doi.org/10.1093/jamia/ocx145 arXiv:/oup/backfile/content_public/journal/jamia/25/3/10.1093_jamia_ocx145/2/ocx145.pdf.
    Locate open access versionFindings
  • [78] H. Suominen, L. Hanlen, C. Paris, Twitter for health — seeking to understand and curate laypersons’ personal experiences: building a social media search engine to improve search, summarization, and visualization, in: A. Neustein (Ed.), Speech Technology and Natural Language Processing in Medicine and Healthcare, De Gruyter, Berlin, Germany, 2014, pp. 134–174 (Chapter 6).
    Google ScholarLocate open access versionFindings
  • [79] M. Johnson, S. Lapkin, V. Long, P. Sanchez, H. Suominen, J. Basilakis, L. Dawson, A systematic review of speech recognition technology in health care, BMC Med. Inform. Decis. Mak. 14 (2014) 94.
    Google ScholarLocate open access versionFindings
  • [80] L. Goeuriot, L. Kelly, H. Suominen, L. Hanlen, A. Névéol, C. Grouin, J.R.M. Palotti, G. Zuccon, Overview of the CLEF eHealth Evaluation Lab 2015, in: J. Mothe, J. Savoy, J. Kamps, K. Pinel-Sauvagnat, G.J.F. Jones, E. SanJuan, L. Cappellato, N. Ferro (Eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction, Proceedings of the Sixth International Conference of the CLEF Association (CLEF 2015), Lecture Notes in Computer Science (LNCS) 9283, Springer, Heidelberg, Germany, 2015, pp. 429–443.
    Google ScholarLocate open access versionFindings
  • [81] T. Hodgson, E. Coiera, Risks and benefits of speech recognition for clinical documentation: a systematic review, J. Am. Med. Inform. Assoc. 23 (e1) (2016) e169–e179.
    Google ScholarLocate open access versionFindings
  • [82] H. Suominen, M. Johnson, L. Zhou, P. Sanchez, R. Sirel, J. Basilakis, L. Hanlen, D. Estival, L. Dawson, B. Kelly, Capturing patient information at nursing shift changes: methodological evaluation of speech recognition and information extraction, J. Am. Med. Inform. Assoc. 22 (e1) (2015) e48–66.
    Google ScholarLocate open access versionFindings
  • [83] T. Hodgson, F. Magrabi, E. Coiera, Evaluating the usability of speech recognition to create clinical documentation using a commercial electronic health record, Int. J. Med. Inform. 113 (2018) 38–42, https://doi.org/10.1016/j.ijmedinf.2018.02.011 <http://www.sciencedirect.com/science/article/pii/S1386505618300820>.
    Locate open access versionFindings
  • [84] D. Mollá, B. Hutchinson, Intrinsic versus extrinsic evaluations of parsing systems, Proceedings of the EACL 2003 Workshop on Evaluation Initiatives in Natural Language Processing: Are Evaluation Methods, Metrics and Resources Reusable?, Evalinitiatives ’03, Association for Computational Linguistics, Stroudsburg, PA, USA, 2003, pp. 43–50 <http://dl.acm.org/citation.cfm?id=1641396.1641403>.
    Locate open access versionFindings
  • [85] K. Nguyen, B. O’Connor, Posterior calibration and exploratory analysis for natural language processing models, Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Lisbon, Portugal, 2015, pp. 1587–1598 <http://aclweb.org/anthology/D15-1182>.
    Locate open access versionFindings
  • [86] W. Scuba, M. Tharp, D. Mowery, E. Tseytlin, Y. Liu, F.A. Drews, W.W. Chapman, Knowledge Author: facilitating user-driven, domain content development to support clinical information extraction, J. Biomed. Semant. 7 (1) (2016) 42, https://doi.org/10.1186/s13326-016-0086-9.
    Locate open access versionFindings
  • [87] J.P.A. Ioannidis, Why most clinical research is not useful, PLOS Med. 13 (6) (2016) 1–10, https://doi.org/10.1371/journal.pmed.1002049.
    Locate open access versionFindings
  • [88] J.P.A. Ioannidis, How to make more published research true, PLoS Med. 11 (10) (2014) e1001747, https://doi.org/10.1371/journal.pmed.1001747.
    Locate open access versionFindings
  • [89] E. von Elm, D.G. Altman, M. Egger, S.J. Pocock, P.C. Gotzsche, J.P. Vandenbroucke, The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies, Lancet (London, England) 370 (9596) (2007) 1453–1457, https://doi.org/10.1016/S0140-6736(07)61602-X.
    Findings
  • [90] E.I. Benchimol, L. Smeeth, A. Guttmann, K. Harron, D. Moher, I. Petersen, H.T. Sørensen, E. von Elm, S.M. Langan, R.W. Committee, The reporting of studies conducted using observational routinely-collected health data (record) statement, PLOS Med. 12 (10) (2015) 1–22, https://doi.org/10.1371/journal.pmed.1001885.
    Locate open access versionFindings
  • [91] R. Gilbert, R. Lafferty, G. Hagger-Johnson, K. Harron, L.-C. Zhang, P. Smith, C. Dibben, H. Goldstein, Guild: Guidance for information about linking data sets, J. Public Health 40 (1) (2018) 191–198, https://doi.org/10.1093/pubmed/fdx037 arXiv:/oup/backfile/content_public/journal/jpubhealth/40/1/10.1093_pubmed_fdx037/3/fdx037.pdf.
    Locate open access versionFindings
Full Text
Your rating :
0

 

Tags
Comments