INTRODUCTION:Automatically identifying specific phenotypes in free-text clinical notes is critically important for the reuse of clinical data. In this study, the authors combine expert-guided feature (text) selection with one-class classification for text processing.OBJECTIVES:To compare the performance of one-class classification to traditional binary classification; to evaluate the utility of feature selection based on expert-selected salient text (snippets); and to determine the robustness of these models with respects to irrelevant surrounding text.METHODS:The authors trained one-class support vector machines (1C-SVMs) and two-class SVMs (2C-SVMs) to identify notes discussing breast cancer. Manually annotated visit summary notes (88 positive and 88 negative for breast cancer) were used to compare the performance of models trained on whole notes labeled as positive or negative to models trained on expert-selected text sections (snippets) relevant to breast cancer status. Model performance was evaluated using a 70:30 split for 20 iterations and on a realistic dataset of 10 000 records with a breast cancer prevalence of 1.4%.RESULTS:When tested on a balanced experimental dataset, 1C-SVMs trained on snippets had comparable results to 2C-SVMs trained on whole notes (F = 0.92 for both approaches). When evaluated on a realistic imbalanced dataset, 1C-SVMs had a considerably superior performance (F = 0.61 vs. F = 0.17 for the best performing model) attributable mainly to improved precision (p = .88 vs. p = .09 for the best performing model).CONCLUSIONS:1C-SVMs trained on expert-selected relevant text sections perform better than 2C-SVMs classifiers trained on either snippets or whole notes when applied to realistically imbalanced data with low prevalence of the positive class.

Medication reconciliation is an important and complex task for which careful user interface design has the potential to help reduce errors and improve quality of care. In this paper we focus on the hospital discharge scenario and first describe a novel interface called Twinlist. Twinlist illustrates the novel use of spatial layout combined with multi-step animation, to help medical providers see what is different and what is similar between the lists (e.g., intake list and hospital list), and rapidly choose the drugs they want to include in the reconciled list. We then describe a series of variant designs and discuss their comparative advantages and disadvantages. Finally we report on a pilot study that suggests that animation might help users learn new spatial layouts such as the one used in Twinlist.

Clinical databases may contain several records for a single patient. Multiple general entity-resolution algorithms have been developed to identify such duplicate records. To achieve optimal accuracy, algorithm parameters must be tuned to a particular dataset. The purpose of this study was to determine the required training set size for probabilistic, deterministic and Fuzzy Inference Engine (FIE) algorithms with parameters optimized using the particle swarm approach. Each algorithm classified potential duplicates into: definite match, non-match and indeterminate (i.e., requires manual review). Training sets size ranged from 2,000-10,000 randomly selected record-pairs. We also evaluated marginal uncertainty sampling for active learning. Optimization reduced manual review size (Deterministic 11.6% vs. 2.5%; FIE 49.6% vs. 1.9%; and Probabilistic 10.5% vs. 3.5%). FIE classified 98.1% of the records correctly (precision=1.0). Best performance required training on all 10,000 randomly-selected record-pairs. Active learning achieved comparable results with 3,000 records. Automated optimization is effective and targeted sampling can reduce the required training set size.

1School of Biomedical Informatics, The University of Texas Health Science Center at Houston, TX; 2National Center for Cognitive Informatics and Decision Making in Healthcare, Houston, TX; 3 The University of Texas MD Anderson Cancer Center. Houston, TX, 77030, USA. 4 FIT Lab — Interaction Laboratory, Swansea University, Swansea, Wales, UK. Eliz A. Markowitz1,2 MS, MS, Todd R. Johnson 1,2 PhD, Elmer V. Bernstam 1,2 MD, MSE, Jorge R. Herskovic 2,3 MD, PhD, Harold Thimbleby 4 PhD A Systematic Yet Flexible Systems Analysis Framework

Introduction Clinical databases require accurate entity resolution (ER). One approach is to use algorithms that assign questionable cases to manual review. Few studies have compared the performance of common algorithms for such a task. Furthermore, previous work has been limited by a lack of objective methods for setting algorithm parameters. We compared the performance of common ER algorithms: using algorithmic optimization, rather than manual parameter tuning, and on two-threshold classification (match/manual review/non-match) as well as single-threshold (match/non-match).Methods We manually reviewed 20 000 randomly selected, potential duplicate record-pairs to identify matches (10 000 training set, 10 000 test set). We evaluated the probabilistic expectation maximization, simple deterministic and fuzzy inference engine (FIE) algorithms. We used particle swarm to optimize algorithm parameters for a single and for two thresholds. We ran 10 iterations of optimization using the training set and report averaged performance against the test set.Results The overall estimated duplicate rate was 6%. FIE and simple deterministic algorithms allowed a lower manual review set compared to the probabilistic method (FIE 1.9%, simple deterministic 2.5%, probabilistic 3.6%; p<0.001). For a single threshold, the simple deterministic algorithm performed better than the probabilistic method (positive predictive value 0.956 vs 0.887, sensitivity 0.985 vs 0.887, p<0.001). ER with FIE classifies 98.1% of record-pairs correctly (1/10 000 error rate), assigning the remainder to manual review.Conclusions Optimized deterministic algorithms outperform the probabilistic method. There is a strong case for considering optimized deterministic methods for ER.

Identifying students at risk for poor performance in large online classes can be challenging. We determined whether the first few quiz scores can be used to identify students who will have poor course outcomes in an introductory informatics class. Mean scores on the first four quizzes can identify students at risk for failure. Even the first quiz score significantly predicted introductory informatics course outcome. Automating early identification of students likely to fail may allow instructors to create targeted interventions. Introduction: Distance learning enabled by the Web is one approach to extend the geographic reach of existing informatics training. Regardless of their career goals, many students start with an introductory informatics course. As a result, more introductory informatics courses are offered online to ever increasing numbers of students with diverse backgrounds and career goals. As the number of students enrolled in each course increases it becomes more difficult to offer personal assistance to every student. If “at risk” students can be automatically identified early, instructors may be able to intervene and rescue students who otherwise would earn a failing grade or drop the course. We determined that the first four weekly quizzes could be used to identify students at risk for failing or dropping an introductory informatics course within the first month of the semester. Methods: We reviewed data from two online introductory informatics courses presented at the University of Texas School of Biomedical Informatics (UT) at Houston and the Medical Informatics program at the University of West Florida (UWF). For UT we used all students beginning with Spring 2007 and ending with Spring 2010, for a total of 205 graduate students. For UWF we used the Fall 2009 semester with a total of 42 students, 28 undergraduates and 14 graduates. We built the prediction model based on UT data and then validated the model using UWF data. We compared automated identification of students at risk for failure to manual identification by a faculty member familiar with the course design and content (TRJ), but not aware of the individual student outcomes. Results: We considered students who scored an average of 75% or below on the weekly quizzes to be at risk of NSC. We chose 75% as the quiz score threshold because this was the highest score where the predictor’s false positives were equal to its false negatives. Even the first quiz score is a significant predictor of course outcome. Each successive quiz added to the model improved the model’s performance. ROC = Receiver Operator Characteristic, PPV/NPV = Positive/Negative Predictive Value. Discussion: While representing only 6% to 8% of the total grade, the first four quizzes are highly predictive for course outcome. Using only the first two quizzes available by the UT add/drop deadline still allows prediction, but with a lower PPV. Conclusion: A simple threshold model developed at UT predicted non-successful completion at another institution offering an introductory course, UWF. Automated prediction compares favorably to human instructor prediction. Acknowledgements: The authors thank all instructors and students who contributed data to this study, including Stephanie Reedy at UWF for help with data collection. This work was funded in part by NCATS Grant UL1 TR000371 establishing the Center for Clinical and Translational Sciences at the University of Texas at Houston. Week(s) Area under ROC Curve (AUC) PPV

OBJECTIVE:To determine whether past access to biomedical documents can predict future document access.MATERIALS AND METHODS:The authors used 394 days of query log (August 1, 2009 to August 29, 2010) from PubMed users in the Texas Medical Center, which is the largest medical center in the world. The authors evaluated two document access models based on the work of Anderson and Schooler. The first is based on how frequently a document was accessed. The second is based on both frequency and recency.RESULTS:The model based only on frequency of past access was highly correlated with the empirical data (R²=0.932), whereas the model based on frequency and recency had a much lower correlation (R²=0.668).DISCUSSION:The frequency-only model accurately predicted whether a document will be accessed based on past use. Modeling accesses as a function of frequency requires storing only the number of accesses and the creation date for the document. This model requires low storage overheads and is computationally efficient, making it scalable to large corpora such as MEDLINE.CONCLUSION:It is feasible to accurately model the probability of a document being accessed in the future based on past accesses.

OBJECTIVETo present a framework for combining implicit knowledge acquisition from multiple experts with machine learning and to evaluate this framework in the context of anemia alerts.MATERIALS AND METHODSFive internal medicine residents reviewed 18 anemia alerts, while 'talking aloud'. They identified features that were reviewed by two or more physicians to determine appropriate alert level, etiology and treatment recommendation. Based on these features, data were extracted from 100 randomly-selected anemia cases for a training set and an additional 82 cases for a test set. Two staff internists assigned an alert level, etiology and treatment recommendation before and after reviewing the entire electronic medical record. The training set of 118 cases (100 plus 18) and the test set of 82 cases were explored using RIDOR and JRip algorithms.RESULTSThe feature set was sufficient to assess 93% of anemia cases (intraclass correlation for alert level before and after review of the records by internists 1 and 2 were 0.92 and 0.95, respectively). High-precision classifiers were constructed to identify low-level alerts (precision p=0.87, recall R=0.4), iron deficiency (p=1.0, R=0.73), and anemia associated with kidney disease (p=0.87, R=0.77).DISCUSSIONIt was possible to identify low-level alerts and several conditions commonly associated with chronic anemia. This approach may reduce the number of clinically unimportant alerts. The study was limited to anemia alerts. Furthermore, clinicians were aware of the study hypotheses potentially biasing their evaluation.CONCLUSIONImplicit knowledge acquisition, collaborative filtering and machine learning were combined automatically to induce clinically meaningful and precise decision rules.

The need to maintain accessibility of the biomedical literature has led to development of methods to assist human indexers by recommending index terms for newly encountered articles. Given the rapid expansion of this literature, it is essential that these methods be scalable. Document vector representations are commonly used for automated indexing, and Random Indexing (RI) provides the means to generate them efficiently. However, RI is difficult to implement in real-world indexing systems, as (1) efficient nearest-neighbor search requires retaining all document vectors in RAM, and (2) it is necessary to maintain a store of randomly generated term vectors to index future documents. Motivated by these concerns, this paper documents the development and evaluation of a deterministic binary variant of RI. The increased capacity demonstrated by binary vectors has implications for information retrieval, and the elimination of the need to retain term vectors facilitates distributed implementations, enhancing the scalability of RI.

Mirroring successes in other industries, efforts to improve health care quality have led to an increased push to develop and adopt systems that enforce or encourage consistent procedures based on best practices and evidence-based medicine. Although such systems can lead to more efficient and safer care, health care is filled with complexity, variations, and exceptions that are not easily captured by idealized processes. Information systems that are too rigid to support such deviations can lead to decreases in quality, along with caregiver resistance and creative workarounds, that together lower the adoption rate and decrease the positive effects of technology [1-5]. Similar problems in other industries have led to the concept of Systematic Yet Flexible (SYF) systems, in which the system supports and encourages a systematic approach, while simultaneously allowing for considerable flexibility [6]. Building upon the general design goals for SYF systems, we designed a framework, called SYFSA (Systematic Yet Flexible System Analysis) for analyzing and designing SYF systems by considering the trade-off between systematicity and flexibility. SYFSA is based on analyzing a task using three related problem spaces: the idealized space, which represents the best practice. the natural space, which captures the possible path actions that may be taken in the real world, and the system space, which specifies how the task is to be done in the the system being (re)designed. According to this framework, the most effective system is one that maximizes the chance of idealized performance, as specified by the task space, while also allowing flexible behavior to cope with the inevitable variations in healthcare tasks. We can use SYFSA to predict performance by comparing the systematicity and flexibility of the task and the system. The long-term goal of this work is to guide the design of systems that allow graceful degradation from ideal performance on standard cases to more flexible performance for coping with variations. Our unifying hypothesis is that SYSFA will allow us to predict user performance. We will test this hypothesis with the following specific aims: Specific Aim 1: Evaluate the flexibility-compatibility hypothesis. We hypothesize that user performance will improve when the flexibility of the system interface matches the flexibility of the task. We will vary task flexibility and interface flexibility in a 2 × BLOCKIN BLOCKIN 2 (task flexibility × system flexibility) design and measure user error rates, task completion rates, and task completion times. Specific Aim 2: Evaluate the prediction that …

Background: Electronic Health Records aggregated in Clinical Data Warehouses (CDWs) promise to revolutionize Comparative Effectiveness Research and suggest new avenues of research. However, the effectiveness of CDWs is diminished by the lack of properly labeled data. We present a novel approach that integrates knowledge from the CDW, the biomedical literature, and the Unified Medical Language System (UMLS) to perform high-throughput phenotyping. In this paper, we automatically construct a graphical knowledge model and then use it to phenotype breast cancer patients. We compare the performance of this approach to using MetaMap when labeling records.Results: MetaMap's overall accuracy at identifying breast cancer patients was 51.1% (n=428); recall=85.4%, precision=26.2%, and F-1=40.1%. Our unsupervised graph-based high-throughput phenotyping had accuracy of 84.1%; recall=46.3%, precision=61.2%, and F-1=52.8%.Conclusions: We conclude that our approach is a promising alternative for unsupervised high-throughput phenotyping.

Medication errors can result from administration inaccuracies at any point of care and are a major cause for concern. To develop a successful Medication Reconciliation (MR) tool, we believe it necessary to build a Work Domain Ontology (WDO) for the MR process. A WDO defines the explicit, abstract, implementation-independent description of the task by separating the task from work context, application technology, and cognitive architecture. We developed a prototype based upon the WDO and designed to adhere to standard principles of interface design. The prototype was compared to Legacy Health System's and Pre-Admission Medication List Builder MR tools via a Keystroke-Level Model analysis for three MR tasks. The analysis found the prototype requires the fewest mental operations, completes tasks in the fewest steps, and completes tasks in the least amount of time. Accordingly, we believe that developing a MR tool, based upon the WDO and user interface guidelines, improves user efficiency and reduces cognitive load.

Background: As the volume of biomedical text increases exponentially, automatic indexing becomes increasingly important. However, existing approaches do not distinguish central (or core) concepts from concepts that were mentioned in passing. We focus on the problem of indexing MEDLINE records, a process that is currently performed by highly trained humans at the National Library of Medicine (NLM). NLM indexers are assisted by a system called the Medical Text Indexer (MTI) that suggests candidate indexing terms.Objective: To improve the ability of MTI to select the core terms in MEDLINE abstracts. These core concepts are deemed to be most important and are designated as "major headings" by MEDLINE indexers. We introduce and evaluate a graph-based indexing methodology called MEDRank that generates concept graphs from biomedical text and then ranks the concepts within these graphs to identify the most important ones.Methods: We insert a MEDRank step into the MTI and compare MTI's output with and without MEDRank to the MEDLINE indexers' selected terms for a sample of 11,803 PubMed Central articles. We also tested whether human raters prefer terms generated by the MEDLINE indexers, MTI without MEDRank, and MTI with MEDRank for a sample of 36 PubMed Central articles.Results: MEDRank improved recall of major headings designated by 30% over MTI without MEDRank (0.489 vs. 0.376). Overall recall was only slightly (6.5%) higher (0.490 vs. 0.460) as was F-2 (3%, 0.408 vs. 0.396). However, overall precision was 3.9% lower (0.268 vs. 0.279). Human raters preferred terms generated by MTI with MEDRank over terms generated by MTI without MEDRank (by an average of 1.00 more term per article), and preferred terms generated by MTI with MEDRank and the MEDLINE indexers at the same rate.Conclusions: The addition of MEDRank to MTI significantly improved the retrieval of core concepts in MEDLINE abstracts and more closely matched human expectations compared to MTI without MEDRank. In addition, MEDRank slightly improved overall recall and F-2. (C) 2011 Elsevier Ireland Ltd. All rights reserved.

Medication reconciliation is a National Patient Safety Goal (NPSG) from The Joint Commission (TJC) that entails reviewing all medications a patient takes after a health care transition. Medication reconciliation is a resource-intensive, error-prone task, and the resources to accomplish it may not be routinely available. Computer-based methods have the potential to overcome these barriers. We designed and explored a rule-based medication reconciliation algorithm to accomplish this task across different healthcare transitions. We tested our algorithm on a random sample of 94 transitions from the Clinical Data Warehouse at the University of Texas Health Science Center at Houston. We found that the algorithm reconciled, on average, 23.4% of the potentially reconcilable medications. Our study did not have sufficient statistical power to establish whether the kind of transition affects reconcilability. We conclude that automated reconciliation is possible and will help accomplish the NPSG.

Proposal and execution of clinical trials, computation of quality measures and discovery of correlation between medical phenomena are all applications where an accurate count of patients is needed. However, existing sources of this type of patient information, including Clinical Data Warehouses (CDWs) may be incomplete or inaccurate. This research explores applying probabilistic techniques, supported by the MayBMS probabilistic database, to obtain accurate patient counts from a Clinical Data Warehouse containing synthetic patient data.We present a synthetic Clinical Data Warehouse, and populate it with simulated data using a custom patient data generation engine. We then implement, evaluate and compare different techniques for obtaining patients counts.We model billing as a test for the presence of a condition. We compute billing's sensitivity and specificity both by conducting a "Simulated Expert Review" where a representative sample of records are reviewed and labeled by experts, and by obtaining the ground truth for every record.We compute the posterior probability of a patient having a condition through a "Bayesian Chain", using Bayes' Theorem to calculate the probability of a patient having a condition after each visit. The second method is a "one-shot" approach that computes the probability of a patient having a condition based on whether the patient is ever billed for the condition.Our results demonstrate the utility of probabilistic approaches, which improve on the accuracy of raw counts. In particular, the simulated review paired with a single application of Bayes' Theorem produces the best results, with an average error rate of 2.1% compared to 43.7% for the straightforward billing counts.Overall, this research demonstrates that Bayesian probabilistic approaches improve patient counts on simulated patient populations. We believe that total patient counts based on billing data are one of the many possible applications of our Bayesian framework. Use of these probabilistic techniques will enable more accurate patient counts and better results for applications requiring this metric.

Online courses will play a key role in the high-volume Informatics education required to train the personnel that will be necessary to fulfill the health IT needs of the country. Online courses can cause feelings of isolation in students. A common way to address these feelings is to hold synchronous online "chats" for students. Conventional chats, however, can be confusing and impose a high extrinsic cognitive load on their participants that hinders the learning process. In this paper we present a qualitative analysis that shows the causes of this high cognitive load and our solution through the use of a moderated chat system.

e16501 Background: Clinical data stored in electronic medical records may be useful for oncology research. The first step for such research is to identify patients who have been diagnosed with malignancies within large clinical cohorts. Structured data (labels) are often collected for billing purposes. Thus, it is tempting to leverage billing data for research purposes. In contrast, clinical information is generally expressed as unstructured text (e.g., dictations). We determined the accuracy of using billing data and machine-learning based classifiers in identifying patients diagnosed with endometrial and breast malignancies. Methods: We manually created a gold standard of persons who have (cases) and have not (controls) been diagnosed with breast/endometrial malignancies. We then determined the recall (R, aka sensitivity) and precision (P, aka positive predictive value) of billing data and sequential minimal optimization support vector machine-based classifiers with polynomial kernels (SMO-SVM, a type of machine learning) and manual feature reduction at identifying cases within a dataset of approximately 230,000 persons. Results: The endometrial cancer gold standard contained 298 randomly selected records enriched for cases using text word searches (22 cases, 276 controls). ICD-9-CM (billing) diagnosis search yielded R=22.7% and P=88.3%. An SVO-SVM classifier identified endometrial neoplasia cases with R=59.1% and P=92.9%. The breast cancer gold standard contained 111 records selected using keyword search (for “breast cancer”). Of 111 records, there were 44 breast cancer cases and 67 controls (keyword search P=39.6%). Searching the gold standard for the ICD-9-CM code for breast cancer (174.*) yielded R=52.2% and P=92.0%. An SMO-SVM classifier performed better (R=88.6%, P=88.6%). Conclusions: To reuse clinical data for oncology research, we cannot rely on billing data or simple text-word searches of clinical cohorts. Machine learning natural language processing techniques are more accurate than either billing data or text-word searches at identifying patients with malignancies within large data sets. Endometrial Breast Recall Precision Recall Precision Billing data 22.7% 88.3% 52.2% 92.0% SMO-SVM classifier 59.1% 92.9% 88.6% 88.6% No significant financial relationships to disclose.

Clinicians, researchers and members of the general public are increasingly using information technology to cope with the explosion in biomedical knowledge. This chapter describes the purpose of query log analysis in the biomedical domain as well as features of the biomedical domain such as controlled vocabularies (ontologies) and existing infrastructure useful for query log analysis. We focus specifically on MEDLINE, which is the most comprehensive bibliographic database of the world’s biomedical literature, the PubMed interface to MEDLINE, the Medical Subject Headings vocabulary and the Unified Medical Language System. However, the approaches discussed here can also be applied to other query logs. We conclude with a look toward the future of biomedical query log analysis.