Many online health information sources are generic and difficult to understand, but consumers want information to be personalized and understandable. Smartphone health applications (apps) offer personalized information to support health goals and reduce preventable chronic conditions. This study aimed to determine how the HealthyMe/MiSalud personalized app (1) engaged English-speaking African American and Spanish-speaking Hispanic adults, and (2) motivated them to set goals and follow preventive recommendations. Our study adds to the literature on digital health, health information seeking, and prevention. We used a multi-method approach, including community and participatory design principles, to learn about potential African American and Hispanic adult health app users and evaluate the app in two usability tests and a 12-month field test. Ninety-six African American and Hispanic adults downloaded the HealthyMe/MiSalud app and used it for a minimum of 36 weeks. We found they wanted personalized information on core prevention topics, and their health histories and goals affected how they rated topic relevance. African American females ages 18-34 were more likely to save an article aligned with family health history, and African American females aged 35-49, males age 50-64, and African American males overall were more likely to save an article aligned with their health goals. Our study revealed that a prevention app with personalized recommendations can support health information seeking and health literacy. These findings can help app developers, public health practitioners, and researchers when designing apps for groups of varying identities.

Recent claims about the impressive abilities of large language models (LLMs) are often supported by evaluating publicly available benchmarks. Since LLMs train on wide swaths of the internet, this practice raises concerns of data contamination, i.e., evaluating on examples that are explicitly or implicitly included in the training data. Data contamination remains notoriously challenging to measure and mitigate, even with partial attempts like controlled experimentation of training data, canary strings, or embedding similarities. In this work, we conduct the first thorough longitudinal analysis of data contamination in LLMs by using the natural experiment of training cutoffs in GPT models to look at benchmarks released over time. Specifically, we consider two code/mathematical problem-solving datasets, Codeforces and Project Euler, and find statistically significant trends among LLM pass rate vs. GitHub popularity and release date that provide strong evidence of contamination. By open-sourcing our dataset, raw results, and evaluation framework, our work paves the way for rigorous analyses of data contamination in modern models. We conclude with a discussion of best practices and future steps for publicly releasing benchmark in the age of LLMs which train on webscale data.

Disaggregated evaluation is a central task in AI fairness assessment, where the goal is to measure an AI system’s performance across different subgroups defined by combinations of demographic or other sensitive attributes. The standard approach is to stratify the evaluation data across subgroups and compute performance metrics separately for each group. However, even for moderately-sized evaluation datasets, sample sizes quickly get small once considering intersectional subgroups, which greatly limits the extent to which intersectional groups are included in analysis. In this work, we introduce a structured regression approach to disaggregated evaluation that we demonstrate can yield reliable system performance estimates even for very small subgroups. We provide corresponding inference strategies for constructing confidence intervals and explore how goodness-of-fit testing can yield insight into the structure of fairness-related harms experienced by intersectional groups. We evaluate our approach on two publicly available datasets, and several variants of semi-synthetic data. The results show that our method is considerably more accurate than the standard approach, especially for small subgroups, and demonstrate how goodness-of-fit testing helps identify the key factors that drive differences in performance.

We explore the use of Large Language Model (LLM-based) chatbots to power recommender systems. We observe that the chatbots respond poorly when they encounter under-specified requests (e.g., they make incorrect assumptions, hedge with a long response, or refuse to answer). We conjecture that such miscalibrated response tendencies (i.e., conversational priors) can be attributed to LLM fine-tuning using annotators -- single-turn annotations may not capture multi-turn conversation utility, and the annotators' preferences may not even be representative of users interacting with a recommender system. We first analyze public LLM chat logs to conclude that query under-specification is common. Next, we study synthetic recommendation problems with configurable latent item utilities and frame them as Partially Observed Decision Processes (PODP). We find that pre-trained LLMs can be sub-optimal for PODPs and derive better policies that clarify under-specified queries when appropriate. Then, we re-calibrate LLMs by prompting them with learned control messages to approximate the improved policy. Finally, we show empirically that our lightweight learning approach effectively uses logged conversation data to re-calibrate the response strategies of LLM-based chatbots for recommendation tasks.

Restless and collapsing bandits are often used to model budget-constrained resource allocation in settings where arms have action-dependent transition probabilities, such as the allocation of health interventions among patients. However, SOTA Whittle-index-based approaches to this planning problem either do not consider fairness among arms, or incentivize fairness without guaranteeing it. We thus introduce ProbFair, a probabilistically fair policy that maximizes total expected reward and satisfies the budget constraint while ensuring a strictly positive lower bound on the probability of being pulled at each timestep. We evaluate our algorithm on a real-world application, where interventions support continuous positive airway pressure (CPAP) therapy adherence among patients, as well as on a broader class of synthetic transition matrices. We find that ProbFair preserves utility while providing fairness guarantees.

Recent claims about the impressive abilities of large language models (LLMs) are often supported by evaluating publicly available benchmarks. Since LLMs train on wide swaths of the internet, this practice raises concerns of data contamination, i.e., evaluating on examples that are explicitly or implicitly included in the training data. Data contamination remains notoriously challenging to measure and mitigate, even with partial attempts like controlled experimentation of training data, canary strings, or embedding similarities. In this work, we conduct the first thorough longitudinal analysis of data contamination in LLMs by using the natural experiment of training cutoffs in GPT models to look at benchmarks released over time. Specifically, we consider two code/mathematical problem-solving datasets, Codeforces and Project Euler, and find statistically significant trends among LLM pass rate vs. GitHub popularity and release date that provide strong evidence of contamination. By open-sourcing our dataset, raw results, and evaluation framework, our work paves the way for rigorous analyses of data contamination in modern models. We conclude with a discussion of best practices and future steps for publicly releasing benchmarks in the age of LLMs that train on webscale data.

Restless multi-armed bandits are often used to model budget-constrained resource allocation tasks where receipt of the resource is associated with an increased probability of a favorable state transition. Prior work assumes that individual arms only benefit if they receive the resource directly. However, many allocation tasks occur within communities and can be characterized by positive externalities that allow arms to derive partial benefit when their neighbor(s) receive the resource. We thus introduce networked restless bandits, a novel multi-armed bandit setting in which arms are both restless and embedded within a directed graph. We then present Greta, a graph-aware, Whittle index-based heuristic algorithm that can be used to efficiently construct a constrained reward-maximizing action vector at each timestep. Our empirical results demonstrate that Greta outperforms comparison policies across a range of hyperparameter values and graph topologies. Code and appendices are available at https://github.com/crherlihy/networked_restless_bandits.

Crowdworker-constructed natural language inference (NLI) datasets have been found to contain statistical artifacts associated with the annotation process that allow hypothesis-only classifiers to achieve better-than-random performance (Poliak et al., 2018; Gururanganet et al., 2018; Tsuchiya, 2018). We investigate whether MedNLI, a physician-annotated dataset with premises extracted from clinical notes, contains such artifacts (Romanov and Shivade, 2018). We find that entailed hypotheses contain generic versions of specific concepts in the premise, as well as modifiers related to responsiveness, duration, and probability. Neutral hypotheses feature conditions and behaviors that co-occur with, or cause, the condition(s) in the premise. Contradiction hypotheses feature explicit negation of the premise and implicit negation via assertion of good health. Adversarial filtering demonstrates that performance degrades when evaluated on the difficult subset. We provide partition information and recommendations for alternative dataset construction strategies for knowledge-intensive domains.

Scientists construct and analyze computational models to understand the world. That understanding comes from efforts to augment, combine, and compare models of related phenomena. We propose SemanticModels.jl, a system that leverages techniques from static and dynamic program analysis to process executable versions of scientific models to perform such metamodeling tasks. By framing these metamodeling tasks as metaprogramming problems, SemanticModels.jl enables writing programs that generate and expand models. To this end, we present a category theory-based framework for defining metamodeling tasks, and extracting semantic information from model implementations, and show how this framework can be used to enhance scientific workflows in a working case study.

SARS-CoV-2, the virus that causes coronavirus disease 2019 (COVID-19), is thought to spread from person to person primarily by the respiratory route and mainly through close contact (1). Community mitigation strategies can lower the risk for disease transmission by limiting or preventing person-to-person interactions (2). U.S. states and territories began implementing various community mitigation policies in March 2020. One widely implemented strategy was the issuance of orders requiring persons to stay home, resulting in decreased population movement in some jurisdictions (3). Each state or territory has authority to enact its own laws and policies to protect the public's health, and jurisdictions varied widely in the type and timing of orders issued related to stay-at-home requirements. To identify the broader impact of these stay-at-home orders, using publicly accessible, anonymized location data from mobile devices, CDC and the Georgia Tech Research Institute analyzed changes in population movement relative to stay-at-home orders issued during March 1-May 31, 2020, by all 50 states, the District of Columbia, and five U.S. territories.* During this period, 42 states and territories issued mandatory stay-at-home orders. When counties subject to mandatory state- and territory-issued stay-at-home orders were stratified along rural-urban categories, movement decreased significantly relative to the preorder baseline in all strata. Mandatory stay-at-home orders can help reduce activities associated with the spread of COVID-19, including population movement and close person-to-person contact outside the household.

SemanticModels.jl is a package that takes advantage of metamodeling and meta-programming to automate model augmentation and creation. We chose Julia for our project because of the powerful type system, the efficient internal abstract syntax tree (AST), and the embedded domain specific languages (DSL) that emerge as a result through the multiple dispatch mechanism for all libraries in the Julia ecosystem [1]. The dynamic type inference and method generation has significant and powerful downstream effects which enables dynamic model manipulation with efficient code generation. These DSLs allow for strong theoretical category definitions and classifications of models to define conceptually sound universal rewrite rules for any given model class.

The COVID-19 Open Research Dataset (CORD-19), a fast growing collection of biomedical research literature, was made available in March 2020 as an effort to facilitate research addressing the COVID-19 pandemic, along with 17 tasks specifying the research questions of interest. We propose an automated, task-informed document retrieval framework for CORD-19 that leverages latent factors learned through topic models to select a set of research articles most relevant to the tasks at hand. Compared to naı̈ve keywordbased approaches, our approach broadens the qualification of relevance from the presence of specific terms to the activity of latent topics. We show that our approach provides an overlapping yet notably different set of selections, as the latent factors account for meaningful document-wise co-occurrences that individual keywords fail to capture. Upon both qualitative and quantitative examination of retrieval results, we further provide recommendations regarding task creation intended for unsupervised document retrieval in large, heterogeneous, natural language datasets.

This project uses deep convolutional neural networks (CNN) to: (1) detect and (2) localize the 14 thoracic pathologies present in the NIH Chest X-ray dataset. Our approach is to develop a multi-label CNN using the network architectures from previous works as baselines, to detect and visualize the selected diseases. We also use a simpler feed forward network to incorporate patient information and compare results against the CNN. Our project presentation slides are available here. Our video presentation is available here. The code is available on Github here

We use a regression discontinuity design to investigate the effect of the Community Reinvestment Act (CRA) on consumer credit outcomes, using data from the Federal Reserve Bank of New York’s Consumer Credit Panel database (hereafter, CCP/Equifax data) for the years 2004 to 2012. A bank’s activities in census tracts with Median Family Incomes (MFIs) that are less than 80 percent of the metropolitan statistical area (MSA) MFI count toward a lending institution’s compliance with CRA rules. Discontinuous changes in consumer credit outcomes at this threshold—assuming census tracts with MFIs at 79.9 percent of the MSA median are the same as census tracts at 80.0 percent, except for CRA eligibility—are evidence of the CRA’s impact. We find no statistically significant effects of the CRA on mortgages or foreclosures, either before or after the financial crisis. We do, however, find evidence that the CRA expanded broad measures of credit market activity; at the CRA threshold, the total number of loans increased 9 percent, the number of people covered by the CCP/Equifax data increased, and the fraction of individuals with a valid risk score increased. Despite expanded credit activity, which may increase consumers’ risk for adverse outcomes, delinquencies did not significantly increase, nor did credit risk score worsen at the CRA threshold.

[ILLUSTRATION OMITTED] Primary sources, as defined by the Library of Congress, are the raw materials of history--original documents and objects which were created at the time under study (Library of Congress, 2014, p. 1). Primary sources differ from secondary sources, which interpret events without the benefit of firsthand experience. While often employed in English, history, and social science courses, primary sources also can be used in science classes to help students acquire knowledge and become more engaged in their coursework, while supporting Next Generation Science Standards (NGSS Lead States 2013), Common Core State Standards (NGAC and CCSSO 2010), and 21st-century skills (Partnership for 21st Century Skills 2014). This article describes a framework called SOURCES (Waring 2014; Waring and Scheiner-Fisher 2014), which is an acronym for a series of steps teachers can follow to integrate primary sources in the science classroom. The unit we offer as an example has students address the question of whether Earthlings are alone in the universe. While this activity, which can be completed in five days, centers on Earth and space science, the method can easily be adapted for use in any science discipline. Using digital resources from the Library of Congress Among online repositories of primary sources, one of the most valuable is the Library of Congress. Among the LOC's digital collections is Finding Our Place in the Cosmos: From Galileo to Sagan and Beyond (see On the web). This online collection of artifacts illustrates how the human perspective of the universe, including ideas about extraterrestrial life, has changed over time. The collection includes perspectives of noted scientists such as Carl Sagan, an astrophysicist, writer, television personality, and professor who simplified astronomy, science, and math for the everyday person to understand. The artifacts include audio recordings, video and movie clips, manuscripts, books, notebooks, lectures, newspaper articles, sheet music, and more. These resources can help educators as they create integrated curricula that students can follow to address such questions as Are we alone in the universe? Learners can explore differing historical perspectives and then postulate and defend their own hypothesis or thesis statement based on available sources from the collection. In general, the collection is appropriate for use in middle grades through high school, though some manuscripts and notebooks have concepts and computations understandable only to students with advanced skills; however, with teacher scaffolding, even these items may still help novice learners understand perspectives on life beyond Earth without actually comprehending the computations. The vast collection may overwhelm some students, so the teacher may wish to provide these students with only selected parts of the collection to assess and analyze. Students may wish to work in groups with the teacher using the SOURCES framework to structure the inquiry process. The SOURCES framework The SOURCES framework for using primary sources has seven steps: 1. Scrutinize the fundamental source 2. Organize thoughts 3. Understand the context 4. Read between the lines 5. Corroborate and refute 6. Establish a plausible narrative 7. Summarize final thoughts 1. Scrutinize the fundamental source Before beginning instruction, the teacher should select a primary source that could be considered fundamental to the learners' understanding of Carl Sagan's perspective concerning our place in the universe and the possibility of extraterrestrial life. We have chosen for our fundamental source an audio recording of Sagan titled the Pale Blue Dot (see On the web), which is also the title of a Sagan book. This four-and-a-half-minute recording presents Sagan's thoughts about the Voyager spacecraft and the place of humans in the universe. …