The European green crab (Carcinus maenas) is a globally important invasive species. Since 1989, it has been spreading north from San Francisco Bay on the Pacific Coast of North America. Climate change is likely to lead to the species spreading further north into Alaskan and Arctic waters. Previous studies predicting its potential future distribution have focused on its range at the end of the century, under only one of multiple climate change scenarios. We used Bayesian Additive Regression Trees (BART) and benthic environmental variable predictors derived from the latest Intergovernmental Panel on Climate Change report to produce decadal predictions under four climate change scenarios; SSP1, SSP2, SSP4, and SSP5. We find that under all but the lowest emission scenarios, the European green crab will find suitable habitat as far north as the Seward Peninsula, and potentially, under the highest emission scenarios, into the Arctic Circle. This could have devastating consequences for the local ecology of the Pacific Coast of North America, for local economies dependent on fisheries, and for local communities, particularly indigenous communities. Increased monitoring in regions where green crab is likely to find favourable habitat could help with early detection, and potentially early functional eradication before populations establish themselves.
Urinary incontinence (UI) affects over half of adult women but is widely underdiagnosed and undertreated in primary care due to stigma, normalization, and limited routine screening. Identifying effective recruitment strategies is essential to improving UI care in these settings. We conducted semi-structured interviews and focus groups with grantee leaders and practice facilitators (n = 32), site visits and virtual interviews with providers and staff (n = 47) and collected quarterly quantitative recruitment and retention data from five grantees participating in a nationwide UI initiative. Qualitative data were analyzed thematically using the Consolidated Framework for Implementation Research. Quantitative data, including 1950 enrolled practices and 134,852 screened patients, were summarized with descriptive statistics. No single recruitment strategy was universally effective. All grantees utilized approaches such as leveraging professional relationships, engaging practice facilitators, and conducting in-person meetings, which were consistently linked to higher practice and provider recruitment. Leadership engagement facilitated recruitment success, while use of incentives and broad outreach alone were insufficient. Patient recruitment was most successful when UI screening was embedded in routine intake workflows. Across all grantees, 21
We investigate whether large language models (LLMs) can predict whether they will succeed on a given task and whether their predictions improve as they progress through multi-step tasks. We also investigate whether LLMs can learn from in-context experiences to make better decisions about whether to pursue a task in scenarios where failure is costly. All LLMs we tested are overconfident, but most predict their success with better-than-random discriminatory power. We find that newer and larger LLMs generally do not have greater discriminatory power, though Claude models do show such a trend. On multi-step agentic tasks, the overconfidence of several frontier LLMs worsens as they progress through the tasks, and reasoning LLMs perform comparably to or worse than non-reasoning LLMs. With in-context experiences of failure, some but not all LLMs reduce their overconfidence leading to significantly improved decision making, while others do not. Interestingly, all LLMs' decisions are approximately rational given their estimated probabilities of success, yet their overly-optimistic estimates result in poor decision making. These results suggest that current LLM agents are hindered by their lack of awareness of their own capabilities. We discuss the implications of LLMs' awareness of their capabilities for AI misuse and misalignment risks.
We present the Judge Reliability Harness, an open source library for constructing validation suites that test the reliability of LLM judges. As LLM based scoring is widely deployed in AI benchmarks, more tooling is needed to efficiently assess the reliability of these methods. Given a benchmark dataset and an LLM judge configuration, the harness generates reliability tests that evaluate both binary judgment accuracy and ordinal grading performance for free-response and agentic task formats. We evaluate four state-of-the-art judges across four benchmarks spanning safety, persuasion, misuse, and agentic behavior, and find meaningful variation in performance across models and perturbation types, highlighting opportunities to improve the robustness of LLM judges. No judge that we evaluated is uniformly reliable across benchmarks using our harness. For example, our preliminary experiments on judges revealed consistency issues as measured by accuracy in judging another LLM's ability to complete a task due to simple text formatting changes, paraphrasing, changes in verbosity, and flipping the ground truth label in LLM-produced responses. The code for this tool is available at: https://github.com/RANDCorporation/judge-reliability-harness
Motivated by the study of state opioid policies, we propose a novel approach that uses autoregressive models for causal effect estimation in settings with panel data and staggered treatment adoption. Specifically, we seek to estimate the impact of key opioid-related policies by quantifying the effects of must access prescription drug monitoring programs (PDMPs), naloxone access laws (NALs), and medical marijuana laws on opioid prescribing. Existing methods, such as differences-in-differences and synthetic controls, are challenging to apply in these types of dynamic policy landscapes where multiple policies are implemented over time and sample sizes are small. Autoregressive models are an alternative strategy that have been used to estimate policy effects in similar settings, but until this paper have lacked formal justification. We outline a set of assumptions that tie these models to causal effects, and we study biases of estimates based on this approach when key causal assumptions are violated. In a set of simulation studies that mirror the structure of our application, we show that our proposed estimators frequently outperform existing estimators. In short, we justify the use of autoregressive models to evaluate the effectiveness of four state policies in combating the opioid crisis.
