The Data Science Academy (DSA) is an extra-curricular program for 7-12th graders that has evolved over the last four years with valuable lessons learned along the way. DSA was created in part because the K-12 curriculum is already packed, so deploying this informal learning environment and outreach program was one way to meet the challenge of the CS4ALL initiative and broaden participation in computing. DSA serves multiple purposes in that regard: it teaches teachers, it gives undergraduates mentoring experience, and it provides a platform for educational research and development. Currently, the DSA comprises five teaching modules, which have been repackaged and delivered as quarter or semester long weekend sessions, or shorter intensive summer programs. That is to say, the DSA accommodates flexible formats, including virtual or in-person ones and soon asynchronous options as well. The DSA provides an opportunity to develop and test lesson modules, including those derived from research projects in partnership with DS-PATH participants (an NSF project to create DS Pathways). Ultimately, the goal is to apply our experience with the DSA in order to expand the K-12 curriculum with new Data Science courses (grades 9-12), modules and pallets (grades 6-8), which is the next phase of our project.

Background and Aims Proton pump inhibitors (PPIs) have been associated with adverse clinical outcomes amongst clopidogrel users after an acute coronary syndrome. Recent pre-clinical results suggest that this risk might extend to subjects without any prior history of cardiovascular disease. We explore this potential risk in the general population via data-mining approaches. Methods Using a novel approach for mining clinical data for pharmacovigilance, we queried over 16 million clinical documents on 2.9 million individuals to examine whether PPI usage was associated with cardiovascular risk in the general population. Results In multiple data sources, we found gastroesophageal reflux disease (GERD) patients exposed to PPIs to have a 1.16 fold increased association (95% CI 1.09–1.24) with myocardial infarction (MI). Survival analysis in a prospective cohort found a two-fold (HR = 2.00; 95% CI 1.07–3.78; P = 0.031) increase in association with cardiovascular mortality. We found that this association exists regardless of clopidogrel use. We also found that H2 blockers, an alternate treatment for GERD, were not associated with increased cardiovascular risk; had they been in place, such pharmacovigilance algorithms could have flagged this risk as early as the year 2000. Conclusions Consistent with our pre-clinical findings that PPIs may adversely impact vascular function, our data-mining study supports the association of PPI exposure with risk for MI in the general population. These data provide an example of how a combination of experimental studies and data-mining approaches can be applied to prioritize drug safety signals for further investigation.

Event sequences, such as patients' medical histories or users' sequences of product reviews, trace how individuals progress over time. Identifying common patterns, or progression stages, in such event sequences is a challenging task because not every individual follows the same evolutionary pattern, stages may have very different lengths, and individuals may progress at different rates. In this paper, we develop a model-based method for discovering common progression stages in general event sequences. We develop a generative model in which each sequence belongs to a class, and sequences from a given class pass through a common set of stages, where each sequence evolves at its own rate. We then develop a scalable algorithm to infer classes of sequences, while also segmenting each sequence into a set of stages. We evaluate our method on event sequences, ranging from patients' medical histories to online news and navigational traces from the Web. The evaluation shows that our methodology can predict future events in a sequence, while also accurately inferring meaningful progression stages, and effectively grouping sequences based on common progression patterns. More generally, our methodology allows us to reason about how event sequences progress over time, by discovering patterns and categories of temporal evolution in large-scale datasets of events.

OBJECTIVE:The trade-off between the speed and simplicity of dictionary-based term recognition and the richer linguistic information provided by more advanced natural language processing (NLP) is an area of active discussion in clinical informatics. In this paper, we quantify this trade-off among text processing systems that make different trade-offs between speed and linguistic understanding. We tested both types of systems in three clinical research tasks: phase IV safety profiling of a drug, learning adverse drug-drug interactions, and learning used-to-treat relationships between drugs and indications. MATERIALS:We first benchmarked the accuracy of the NCBO Annotator and REVEAL in a manually annotated, publically available dataset from the 2008 i2b2 Obesity Challenge. We then applied the NCBO Annotator and REVEAL to 9 million clinical notes from the Stanford Translational Research Integrated Database Environment (STRIDE) and used the resulting data for three research tasks. RESULTS:There is no significant difference between using the NCBO Annotator and REVEAL in the results of the three research tasks when using large datasets. In one subtask, REVEAL achieved higher sensitivity with smaller datasets. CONCLUSIONS:For a variety of tasks, employing simple term recognition methods instead of advanced NLP methods results in little or no impact on accuracy when using large datasets. Simpler dictionary-based methods have the advantage of scaling well to very large datasets. Promoting the use of simple, dictionary-based methods for population level analyses can advance adoption of NLP in practice.

Electronic health records (EHR) represent a rich and relatively untapped resource for characterizing the true nature of clinical practice and for quantifying the degree of inter-relatedness of medical entities such as drugs, diseases, procedures and devices. We provide a unique set of co-occurrence matrices, quantifying the pairwise mentions of 3 million terms mapped onto 1 million clinical concepts, calculated from the raw text of 20 million clinical notes spanning 19 years of data. Co-frequencies were computed by means of a parallelized annotation, hashing, and counting pipeline that was applied over clinical notes from Stanford Hospitals and Clinics. The co-occurrence matrix quantifies the relatedness among medical concepts which can serve as the basis for many statistical tests, and can be used to directly compute Bayesian conditional probabilities, association rules, as well as a range of test statistics such as relative risks and odds ratios. This dataset can be leveraged to quantitatively assess comorbidity, drug-drug, and drug-disease patterns for a range of clinical, epidemiological, and financial applications.

A key rate-limiting step in using electronic health records for research is the creation of electronic phenotyping algorithms. It is widely agreed that methods for electronic phenotyping should use the totality of EHR data including clinical notes, laboratory test results and medication orders, besides the coded administrative data that are readily available. In addition to efforts at creating consensus definitions for health outcomes, there are efforts at using machine learning to construct descriptions of phenotypes in lieu of traditional “algorithms” that identify patients with a health outcome of interest. A bottleneck in scaling the use of manually created clinical phenotyping algorithms is the time required in their creation and for the machine learning approaches the bottleneck is the creation of a manually labeled gold standard for training. It is clear that just focusing on manually creating larger training sets is not cost-effective. We demonstrate the feasibility of using large, automatically created ‘silver standards’ from comprehensive EHR data, in conjunction with expert knowledge codified in existing ontologies, to create phenotype models via machine learning. Introduction As electronic health records (EHRs) become available for research, there are a multitude of efforts for identifying sets of patients that correspond to a phenotype of interest (1, 2, 3, 4). Robust descriptions or “algorithms” for phenotypes that can be applied across clinical data warehouses would allow reuse of data already collected in EHRs to advance our understanding of disease, to improve clinical practice, and to make discoveries through clinical data mining (5). Thus, solving the electronic phenotyping problem could address one of the key barriers in clinical research informatics— the effort required for identifying cases and abstracting clinical data (3). Recent advances in EHR-driven phenotyping are breaking down these barriers. (For a review, see (1).) For example, Liao et al (6) showed that regression models trained on labeled data can accurately identify rheumatoid arthritis cases with a PPV of 94%. Carroll et al (7) showed that the coefficients of such models constitute a kind of phenotype description that is portable across sites with high accuracy (AUC 92-97%); and finally, Carrell et al (8) argue in a study on breast cancer recurrence that natural language processing (NLP) techniques could reduce the number of charts reviewed for identifying cases by 90%. Their findings are similar to earlier findings demonstrating the equivalent performance of using NLP-based methods (AUC 0.85) with that of manual approaches (AUC 0.87) (9). Importantly, Carrell et al note that the NLP techniques miss only 8% of cases (5 out of 65), most of which (80%; 4 out of 5) were due to not having documents at all. Several authors argue that the comprehensive use of EHRs outweighs the use of codified data alone and can enable the construction of portable phenotype descriptions (2, 6, 7). By comprehensive, we mean that in addition to billing procedure and diagnostic codes, we include patient demographics, features extracted from clinical notes (such as medically relevant phrases from admission, discharge and progress notes as well as radiology and pathology reports), medication prescriptions, and laboratory data over the entire length of the record. There is general agreement that the rate-limiting step in clinical informatics research is the generation of clinical phenotype descriptions that can be used to build patient cohorts (3) and that just focusing on manually creating larger training sets is not cost-effective (10). If we reduce the time in “assigning clean labels” by settling for automated and possibly less than perfect silver standards, we can leverage the “comprehensiveness and volume” of EHR data to get accurate enough phenotype models that enable discovery as well as advance medical practice (1115). If we pursue a machine-learning approach to identifying the set of patients that constitute a cohort of interest, the problem boils down to having accurately labeled patients, i.e., a sufficiently large set of people with (and without) the phenotype of interest, accompanied with their comprehensive EHR data to be used as features. We report here the analyses and results from two separate experiments. First, by an analysis at Columbia University Medical Center, we demonstrate face validity and feasibility of compiling silver standards by showing that features, most significantly associated with a phenotype identified using descriptive phrases, included features that were a priori expected to be useful in confirming the condition. In the second experiment using data from Stanford

Text mining is the computational process of extracting meaningful information from large amounts of unstructured text. It is emerging as a tool to leverage underutilized data sources that can improve pharmacovigilance, including the objective of adverse drug event (ADE) detection and assessment. This article provides an overview of recent advances in pharmacovigilance driven by the application of text mining, and discusses several data sources—such as biomedical literature, clinical narratives, product labeling, social media, and Web search logs—that are amenable to text mining for pharmacovigilance. Given the state of the art, it appears text mining can be applied to extract useful ADE-related information from multiple textual sources. Nonetheless, further research is required to address remaining technical challenges associated with the text mining methodologies, and to conclusively determine the relative contribution of each textual source to improving pharmacovigilance.

Off-label drug use, defined as use of a drug in a manner that deviates from its approved use defined by the drug's FDA label, is problematic because such uses have not been evaluated for safety and efficacy. Studies estimate that 21% of prescriptions are off-label, and only 27% of those have evidence of safety and efficacy. We describe a data-mining approach for systematically identifying off-label usages using features derived from free text clinical notes and features extracted from two databases on known usage (Medi-Span and DrugBank). We trained a highly accurate predictive model that detects novel off-label uses among 1,602 unique drugs and 1,472 unique indications. We validated 403 predicted uses across independent data sources. Finally, we prioritize well-supported novel usages for further investigation on the basis of drug safety and cost.

Display Omitted We propose a computationally-efficient approach to ontology subset extraction.We use the SNOMED CORE subset as source; NDF-RT, RxNorm, and the UMLS Metathesaurus.We extract a targeted subset of drugs, RxNorm/CORE, 4% the size of RxNorm.Recall was better when annotating scientific abstracts than clinical texts.Ontology repositories with mappings look promising to extract subsets for annotation. The benefits of using ontology subsets versus full ontologies are well-documented for many applications. In this study, we propose an efficient subset extraction approach for a domain using a biomedical ontology repository with mappings, a cross-ontology, and a source subset from a related domain. As a case study, we extracted a subset of drugs from RxNorm using the UMLS Metathesaurus, the NDF-RT cross-ontology, and the CORE problem list subset of SNOMED CT. The extracted subset, which we termed RxNorm/CORE, was 4% the size of the full RxNorm (0.4% when considering ingredients only). For evaluation, we used CORE and RxNorm/CORE as thesauri for the annotation of clinical documents and compared their performance to that of their respective full ontologies (i.e., SNOMED CT and RxNorm). The wide range in recall of both CORE (29-69%) and RxNorm/CORE (21-35%) suggests that more quantitative research is needed to assess the benefits of using ontology subsets as thesauri in annotation applications. Our approach to subset extraction, however, opens a door to help create other types of clinically useful domain specific subsets and acts as an alternative in scenarios where well-established subset extraction techniques might suffer from difficulties or cannot be applied.

The benefits of using ontology subsets versus full ontologies are well-documented for many applications. In this study, we propose an efficient subset extraction approach for a domain using a biomedical ontology repository with mappings, a cross-ontology, and a source subset from a related domain. As a case study, we extracted a subset of drugs from RxNorm using the UMLS Metathesaurus, the NDF-RT cross-ontology, and the CORE problem list subset of SNOMED CT. The extracted subset, which we termed RxNorm/CORE, was 4% the size of the full RxNorm (0.4% when considering ingredients only). For evaluation, we used CORE and RxNorm/CORE as thesauri for the annotation of clinical documents and compared their performance to that of their respective full ontologies (i.e., SNOMED CT and RxNorm). The wide range in recall of both CORE (29-69%) and RxNorm/CORE (21-35%) suggests that more quantitative research is needed to assess the benefits of using ontology subsets as thesauri in annotation applications. Our approach to subset extraction, however, opens a door to help create other types of clinically useful domain specific subsets and acts as an alternative in scenarios where well-established subset extraction techniques might suffer from difficulties or cannot be applied.

OBJECTIVE:Depression is a prevalent disorder difficult to diagnose and treat. In particular, depressed patients exhibit largely unpredictable responses to treatment. Toward the goal of personalizing treatment for depression, we develop and evaluate computational models that use electronic health record (EHR) data for predicting the diagnosis and severity of depression, and response to treatment. MATERIALS AND METHODS:We develop regression-based models for predicting depression, its severity, and response to treatment from EHR data, using structured diagnosis and medication codes as well as free-text clinical reports. We used two datasets: 35,000 patients (5000 depressed) from the Palo Alto Medical Foundation and 5651 patients treated for depression from the Group Health Research Institute. RESULTS:Our models are able to predict a future diagnosis of depression up to 12 months in advance (area under the receiver operating characteristic curve (AUC) 0.70-0.80). We can differentiate patients with severe baseline depression from those with minimal or mild baseline depression (AUC 0.72). Baseline depression severity was the strongest predictor of treatment response for medication and psychotherapy. CONCLUSIONS:It is possible to use EHR data to predict a diagnosis of depression up to 12 months in advance and to differentiate between extreme baseline levels of depression. The models use commonly available data on diagnosis, medication, and clinical progress notes, making them easily portable. The ability to automatically determine severity can facilitate assembly of large patient cohorts with similar severity from multiple sites, which may enable elucidation of the moderators of treatment response in the future.

It is widely anticipated that the full value of data in electronic health records will come from mining clinical notes. In practice, in order to perform data-mining studies, we need to process and analyze more than 10 million notes even for our relatively small population of patients at Stanford. Using wellknown NLP tools, this processing would either take years or require the kind of horsepower that makes logistics, expertise, security, and access a different kind of barrier to overcome. Using a clinical text processing system inspired by the NCBO Annotator and Resource Index, we process over 3 million notes per hour per CPU, generate a patient–feature matrix of nearly a trillion cells, and use this data to answer biomedical questions in four application areas ranging from pharmacovigilance to comparative effectiveness.

BACKGROUND AND OBJECTIVE Electronic health records (EHRs) are increasingly being used to complement the FDA Adverse Event Reporting System (FAERS) and to enable active pharmacovigilance. Over 30% of all adverse drug reactions are caused by drug-drug interactions (DDIs) and result in significant morbidity every year, making their early identification vital. We present an approach for identifying DDI signals directly from the textual portion of EHRs. METHODS We recognize mentions of drug and event concepts from over 50 million clinical notes from two sites to create a timeline of concept mentions for each patient. We then use adjusted disproportionality ratios to identify significant drug-drug-event associations among 1165 drugs and 14 adverse events. To validate our results, we evaluate our performance on a gold standard of 1698 DDIs curated from existing knowledge bases, as well as with signaling DDI associations directly from FAERS using established methods. RESULTS Our method achieves good performance, as measured by our gold standard (area under the receiver operator characteristic (ROC) curve >80%), on two independent EHR datasets and the performance is comparable to that of signaling DDIs from FAERS. We demonstrate the utility of our method for early detection of DDIs and for identifying alternatives for risky drug combinations. Finally, we publish a first of its kind database of population event rates among patients on drug combinations based on an EHR corpus. CONCLUSIONS It is feasible to identify DDI signals and estimate the rate of adverse events among patients on drug combinations, directly from clinical text; this could have utility in prioritizing drug interaction surveillance as well as in clinical decision support.

In biomedical research, network analysis provides a conceptual framework for interpreting data from high-throughput experiments. For example, protein-protein interaction networks have been successfully used to identify candidate disease genes. Recently, advances in clinical text processing and the increasing availability of clinical data have enabled analogous analyses on data from electronic medical records. We constructed networks of diseases, drugs, medical devices and procedures using concepts recognized in clinical notes from the Stanford clinical data warehouse. We demonstrate the use of the resulting networks for clinical research informatics in two ways-cohort construction and outcomes analysis-by examining the safety of cilostazol in peripheral artery disease patients as a use case. We show that the network-based approaches can be used for constructing patient cohorts as well as for analyzing differences in outcomes by comparing with standard methods, and discuss the advantages offered by network-based approaches.

BACKGROUND:Proton pump inhibitors (PPIs) are gastric acid-suppressing agents widely prescribed for the treatment of gastroesophageal reflux disease. Recently, several studies in patients with acute coronary syndrome have raised the concern that use of PPIs in these patients may increase their risk of major adverse cardiovascular events. The mechanism of this possible adverse effect is not known. Whether the general population might also be at risk has not been addressed. METHODS AND RESULTS:Plasma asymmetrical dimethylarginine (ADMA) is an endogenous inhibitor of nitric oxide synthase. Elevated plasma ADMA is associated with increased risk for cardiovascular disease, likely because of its attenuation of the vasoprotective effects of endothelial nitric oxide synthase. We find that PPIs elevate plasma ADMA levels and reduce nitric oxide levels and endothelium-dependent vasodilation in a murine model and ex vivo human tissues. PPIs increase ADMA because they bind to and inhibit dimethylarginine dimethylaminohydrolase, the enzyme that degrades ADMA. CONCLUSIONS:We present a plausible biological mechanism to explain the association of PPIs with increased major adverse cardiovascular events in patients with unstable coronary syndromes. Of concern, this adverse mechanism is also likely to extend to the general population using PPIs. This finding compels additional clinical investigations and pharmacovigilance directed toward understanding the cardiovascular risk associated with the use of the PPIs in the general population.

The current state of the art in post-marketing drug surveillance utilizes voluntarily submitted reports of suspected adverse drug reactions. We present data mining methods that transform unstructured patient notes taken by doctors, nurses and other clinicians into a de-identified, temporally ordered, patient-feature matrix using standardized medical terminologies. We demonstrate how to use the resulting high-throughput data to monitor for adverse drug events based on the clinical notes in the EHR.