Language models can perceive visual properties of text despite receiving only sequences of tokens-we mechanistically investigate how Claude 3.5 Haiku accomplishes one such task: linebreaking in fixed-width text. We find that character counts are represented on low-dimensional curved manifolds discretized by sparse feature families, analogous to biological place cells. Accurate predictions emerge from a sequence of geometric transformations: token lengths are accumulated into character count manifolds, attention heads twist these manifolds to estimate distance to the line boundary, and the decision to break the line is enabled by arranging estimates orthogonally to create a linear decision boundary. We validate our findings through causal interventions and discover visual illusions–character sequences that hijack the counting mechanism. Our work demonstrates the rich sensory processing of early layers, the intricacy of attention algorithms, and the importance of combining feature-based and geometric views of interpretability.

We demonstrate that sparse autoencoders can extract interpretable features from Claude 3 Sonnet, a production-scale language model, addressing the open question of whether dictionary learning methods scale beyond small transformers. We trained sparse autoencoders with up to 34 million features on the model's middle layer residual stream, using scaling laws to guide hyperparameter selection. The resulting features are multilingual and multimodal (generalizing to images despite text-only training), respond to both concrete instances and abstract discussions of concepts, and can be used to steer model behavior in ways consistent with their interpretations. We find features corresponding to famous entities and locations, as well as more abstract concepts like sarcasm or errors in code. We also identify features relevant to ways in which language models might cause harm–including features representing deception, power-seeking, sycophancy, and bias–and show that these causally influence model outputs when manipulated. Additionally, we conduct analyses of feature interpretability, geometry, and computational function. However, significant limitations remain: our suite of features is incomplete, and we lack rigorous methods for evaluating whether our features faithfully capture model computations.

Out of everything the human brain processes, only a small fraction is consciously accessible, in the sense of being available for verbal report, deliberate control, and flexible reasoning. In this paper, we present evidence that an analogous functional distinction has emerged in large language models. Using a new interpretability technique, the Jacobian lens, we identify the representations a model is poised to verbalize at any point in its processing. These representations, which we collectively call the J-space, exhibit the functional properties characteristic of a global workspace: their contents can be reported, deliberately summoned and held, used to carry the intermediate steps of silent reasoning, and passed as arguments to arbitrary downstream computations, while automatic processing such as text parsing and routine inference proceeds without them. The J-space also has structural signatures that global workspace theory associates with conscious access: it carries coherent content only in an intermediate band of layers, holds on the order of tens of concepts at a time, and is broadcast by the model's weights more widely than other representations. These properties make it a practical window into a model's unspoken thinking. In alignment audits, it reveals strategic deliberation, evaluation awareness, and trained-in misaligned dispositions that never appear in the model's outputs. We find that post-training installs the Assistant's point of view in the workspace, and we introduce counterfactual reflection training, which improves behavior by training only what a model would say if interrupted and asked to reflect. These results indicate that language models maintain a small, privileged set of representations bearing some of the functional hallmarks of conscious access, and that decoding these representations sheds light on ongoing cognitive processes.

We demonstrate the emergence of scaling laws in the benchmark top versus QCD jet classification problem in collider physics. Six distinct physically-motivated classifiers exhibit power-law scaling of the binary cross-entropy test loss as a function of training set size, with distinct power law indices. This result highlights the importance of comparing classifiers as a function of dataset size rather than for a fixed training set, as the optimal classifier may change considerably as the dataset is scaled up. We speculate on the interpretation of our results in terms of previous models of scaling laws observed in natural language and image datasets.

Diffusion MRI (dMRI) non-invasively maps brain white matter, yet necessitates denoising due to low signal-to-noise ratios. Patch2Self (P2S), employing self-supervised techniques and regression on a Casorati matrix, effectively denoises dMRI images and has become the new de-facto standard in this field. P2S however is resource intensive, both in terms of running time and memory usage, as it uses all voxels (n) from all-but-one held-in volumes (d- 1) to learn a linear mapping Phi : Rnx(d-1) bar right arrow R-n for denoising the held-out volume. The increasing size and dimensionality of higher resolution dMRI acquisitions can make P2S infeasible for large-scale analyses. This work exploits the redundancy imposed by P2S to alleviate its performance issues and inspect regions that influence the noise disproportionately. Specifically, this study makes a three-fold contribution: (1) We present Patch2Self2 (P2S2), a method that uses matrix sketching to perform self-supervised denoising. By solving a sub-problem on a smaller sub-space, so called, coreset, we show how P2S2 can yield a significant speedup in training time while using less memory. (2) We present a theoretical analysis of P2S2, focusing on determining the optimal sketch size through rank estimation, a key step in achieving a balance between denoising accuracy and computational efficiency. (3) We show how the so-called statistical leverage scores can be used to interpret the denoising of dMRI data, a process that was traditionally treated as a black-box. Experimental results on both simulated and real data affirm that P2S2 maintains denoising quality while significantly enhancing speed and

ABSTRACT Metagenomic studies have uncovered many novel viruses by looking beyond hosts of public health or economic interest. However, the resulting viral genomes are often incomplete, and analyses largely characterize the distribution of viruses over their dynamics. Here, we integrate accumulated data from metagenomic studies to reveal geographic and evolutionary dynamics in a case study of Orthomyxoviridae , the RNA virus family that includes influenza virus. First, we use sequences of the orthomyxovirid Wǔhàn mosquito virus 6 to track the migrations of its host. We then look at orthomyxovirus genome evolution, finding gene gain and loss across members of the family, especially in the surface proteins responsible for cell and host tropism. We find that the surface protein of Wǔhàn mosquito virus 6 exhibits accelerated non-synonymous evolution suggestive of antigenic evolution, i.e., vertebrate infection, and belongs to a wider quaranjavirid group bearing highly diverged surface proteins. Finally, we quantify the progress of orthomyxovirus discovery and forecast that many diverged Orthomyxoviridae members remain to be found. We argue that continued metagenomic studies will be fruitful for understanding the dynamics, evolution, ecology of viruses, and their hosts, regardless of whether novel viruses are identified or not, as long as study designs allowing for the resolution of complete viral genomes are employed. IMPORTANCE The number of known virus species has increased dramatically through metagenomic studies, which search genetic material sampled from a host for non-host genes. Here, we focus on an important viral family that includes influenza viruses, the Orthomyxoviridae , with over 100 recently discovered viruses infecting hosts from humans to fish. We find that one virus called Wǔhàn mosquito virus 6, discovered in mosquitoes in China, has spread across the globe very recently. Surface proteins used to enter cells show signs of rapid evolution in Wǔhàn mosquito virus 6 and its relatives which suggests an ability to infect vertebrate animals. We compute the rate at which new orthomyxovirus species discovered add evolutionary history to the tree of life, predict that many viruses remain to be discovered, and discuss what appropriately designed future studies can teach us about how diseases cross between continents and species.

IMPORTANCE:The number of known virus species has increased dramatically through metagenomic studies, which search genetic material sampled from a host for non-host genes. Here, we focus on an important viral family that includes influenza viruses, the Orthomyxoviridae, with over 100 recently discovered viruses infecting hosts from humans to fish. We find that one virus called Wǔhàn mosquito virus 6, discovered in mosquitoes in China, has spread across the globe very recently. Surface proteins used to enter cells show signs of rapid evolution in Wǔhàn mosquito virus 6 and its relatives which suggests an ability to infect vertebrate animals. We compute the rate at which new orthomyxovirus species discovered add evolutionary history to the tree of life, predict that many viruses remain to be discovered, and discuss what appropriately designed future studies can teach us about how diseases cross between continents and species.

Quantitative diffusion MRI (dMRI) is a promising technique for evaluating the spinal cord in health and dis-ease. However, low signal-to-noise ratio (SNR) can impede interpretation and quantification of these images. The purpose of this study is to evaluate several dMRI denoising approaches on their ability to improve the quality, reliability, and accuracy of quantitative diffusion MRI of the spinal cord. We evaluate three denoising approaches (Non-Local Means, Marchenko-Pastur PCA, and a newly proposed Patch2Self algorithm) and conduct five exper-iments to validate the denoising performance on clinical-quality and commonly-acquired dMRI acquisitions: 1) a phantom experiment to assess denoising error and bias; 2) a multi-vendor, multi-acquisition open experiment for both qualitative and quantitative evaluation of noise residuals; 3) a bootstrapping experiment to estimate uncer-tainty of parametric maps; 4) an assessment of spinal cord lesion conspicuity in a multiple sclerosis group; and 5) an evaluation of denoising for advanced parametric multi-compartment modeling. We find that all methods improve signal-to-noise ratio and conspicuity of MS lesions in individual diffusion weighted images (DWIs), but MPPCA and Patch2Self excel at improving the quality and intra-cord contrast of diffusion weighted images - re-moving signal fluctuations due to thermal noise while improving precision of estimation of diffusion parameters even with very few DWIs (i.e., 16-32) typical of clinical acquisitions. These denoising approaches hold promise for facilitating reliable diffusion observations and measurements in the spinal cord to investigate biological and pathological processes.

During the COVID-19 pandemic within the United States, much of the responsibility for diagnostic testing and epidemiologic response has relied on the action of county-level departments of public health. Here we describe the integration of genomic surveillance into epidemiologic response within Humboldt County, a rural county in northwest California. Through a collaborative effort, 853 whole SARS-CoV-2 genomes were generated, representing ~58% of the 1,449 SARS-CoV-2-positive cases detected in Humboldt County as of March 12, 2021. Phylogenetic analysis of these data was used to develop a comprehensive understanding of SARS-CoV-2 introductions to the county and to support contact tracing and epidemiologic investigations of all large outbreaks in the county. In the case of an outbreak on a commercial farm, viral genomic data were used to validate reported epidemiologic links and link additional cases within the community who did not report a farm exposure to the outbreak. During a separate outbreak within a skilled nursing facility, genomic surveillance data were used to rule out the putative index case, detect the emergence of an independent Spike:N501Y substitution, and verify that the outbreak had been brought under control. These use cases demonstrate how developing genomic surveillance capacity within local public health departments can support timely and responsive deployment of genomic epidemiology for surveillance and outbreak response based on local needs and priorities.

Diffusion MRI typically has a low SNR on account of the noise from a variety of sources corrupting the data. The state-of-the-art denoiser Patch2Self proposed a self-supervised learning technique that uses patches from all the voxels to learn the denoising function which in practice can be resource-intensive. We, therefore, propose Patch2Self2 which performs self-supervised denoising using coresets constructed via matrix sketching, resulting in significant speedups and reduced memory usage. Our results showed that sampling-based sketching via leverage scores gave the best performance. Remarkably, leverage scores can be directly used as a statistic for interpreting influential regions hampering the denoising performance.

BACKGROUND:Outbreaks of SARS-CoV-2 in long-term care facilities (LTCFs) cause significant morbidity and mortality. Mapping viral transmission within and between facilities by combining genomic sequencing with epidemiologic investigations enables targeting infection-control interventions.METHODS:We conducted weekly surveillance of residents and staff in LTCFs in Santa Clara County, California, with ≥1 confirmed COVID-19 case between March and July 2020. Positive samples were referred for whole-genome sequencing. Epidemiological investigations and phylogenetic analyses of the largest outbreaks (>30 cases) were carried out in 6 LTCFs (Facilities A through F).RESULTS:Among the 61 LTCFs in the county, 41 had ≥1 confirmed case during the study period, triggering weekly SARS-CoV-2 testing. The 6 largest outbreaks accounted for 60% of cases and 90% of deaths in LTCFs, although the bed capacity of these facilities represents only 11% of the LTCF beds in the county. Phylogenetic analysis of 196 whole-genome sequences recovered from those facilities showed that each outbreak was monophyletic, with staff and residents sharing a common viral lineage. Outbreak investigations revealed that infected staff members often worked at multiple facilities, and in 1 instance, a staff member infected while working in 1 facility was the likely index case in another.CONCLUSIONS:We detected a pattern of rapid and sustained transmission after a single introduction of SARS-CoV-2 in 6 large LTCF outbreaks, with staff playing a key role in transmission within and between facilities. Infection control, testing, and occupational policies to reduce exposure and transmission risk for staff are essential components to keeping facility residents safe.

There is an urgent need to understand the dynamics and risk factors driving ongoing severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) transmission during shelter-in-place mandates. We offered SARS-CoV-2 reverse-transcription polymerase chain reaction (PCR) and antibody (Abbott ARCHITECT IgG) testing, regardless of symptoms, to all residents (aged ≥4 years) and workers in a San Francisco census tract (population: 5174) at outdoor, community-mobilized events over 4 days. We estimated SARS-CoV-2 point prevalence (PCR positive) and cumulative incidence (antibody or PCR positive) in the census tract and evaluated risk factors for recent (PCR positive/antibody negative) vs prior infection (antibody positive/PCR negative). SARS-CoV-2 genome recovery and phylogenetics were used to measure viral strain diversity, establish viral lineages present, and estimate number of introductions. We tested 3953 persons (40% Latinx; 41% White; 9% Asian/Pacific Islander; and 2% Black). Overall, 2.1% (83/3871) tested PCR positive: 95% were Latinx and 52% were asymptomatic when tested; 1.7% of census tract residents and 6.0% of workers (non–census tract residents) were PCR positive. Among 2598 tract residents, estimated point prevalence of PCR positives was 2.3% (95% confidence interval [CI], 1.2%–3.8%): 3.9% (95% CI, 2.0%–6.4%) among Latinx persons vs 0.2% (95% CI, .0–.4%) among non-Latinx persons. Estimated cumulative incidence among residents was 6.1% (95% CI, 4.0%–8.6%). Prior infections were 67% Latinx, 16% White, and 17% other ethnicities. Among recent infections, 96% were Latinx. Risk factors for recent infection were Latinx ethnicity, inability to shelter in place and maintain income, frontline service work, unemployment, and household income <$50 000/year. Five SARS-CoV-2 phylogenetic lineages were detected. SARS-CoV-2 infections from diverse lineages continued circulating among low-income, Latinx persons unable to work from home and maintain income during San Francisco’s shelter-in-place ordinance.

Mosquitoes are major infectious disease-carrying vectors. Assessment of current and future risks associated with the mosquito population requires knowledge of the full repertoire of pathogens they carry, including novel viruses, as well as their blood meal sources. Unbiased metatranscriptomic sequencing of individual mosquitoes offers a straightforward, rapid, and quantitative means to acquire this information. Here, we profile 148 diverse wild-caught mosquitoes collected in California and detect sequences from eukaryotes, prokaryotes, 24 known and 46 novel viral species. Importantly, sequencing individuals greatly enhanced the value of the biological information obtained. It allowed us to (a) speciate host mosquito, (b) compute the prevalence of each microbe and recognize a high frequency of viral co-infections, (c) associate animal pathogens with specific blood meal sources, and (d) apply simple co-occurrence methods to recover previously undetected components of highly prevalent segmented viruses. In the context of emerging diseases, where knowledge about vectors, pathogens, and reservoirs is lacking, the approaches described here can provide actionable information for public health surveillance and intervention decisions.

Autoencoders have been proposed as a powerful tool for model-independent anomaly detection in high-energy physics. The operating principle is that events which do not belong to the space of training data will be reconstructed poorly, thus flagging them as anomalies. We point out that in a variety of examples of interest, the connection between large reconstruction error and anomalies is not so clear. In particular, for data sets with nontrivial topology, there will always be points that erroneously seem anomalous due to global issues. Conversely, neural networks typically have an inductive bias or prior to locally interpolate such that undersampled or rare events may be reconstructed with small error, despite actually being the desired anomalies. Taken together, these facts are in tension with the simple picture of the autoencoder as an anomaly detector. Using a series of illustrative low-dimensional examples, we show explicitly how the intrinsic and extrinsic topology of the dataset affects the behavior of an autoencoder and how this topology is manifested in the latent space representation during training. We ground this analysis in the discussion of a mock "bump hunt" in which the autoencoder fails to identify an anomalous "signal" for reasons tied to the intrinsic topology of $n$-particle phase space.

We identify a mutation in the N gene of SARS-CoV-2 that adversely affects annealing of a commonly used RT-PCR primer; epidemiologic evidence suggests the virus retains pathogenicity and competence for spread. This reinforces the importance of using multiple targets, preferably in at least 2 genes, for robust SARS-CoV-2 detection. Article Summary Line A SARS-CoV-2 variant that occurs worldwide and has spread in California significantly affects diagnostic sensitivity of an N gene assay, highlighting the need to employ multiple viral targets for detection.

We combined viral genome sequencing with contact tracing to investigate introduction and evolution of severe acute respiratory syndrome coronavirus 2 lineages in Santa Clara County, California, from 27 January to 21 March 2020. From 558 persons with coronavirus disease 2019, 101 genomes from 143 available clinical samples comprised 17 lineages, including SCC1 (n = 41), WA1 (n = 9; including the first 2 reported deaths in the United States, with postmortem diagnosis), D614G (n = 4), ancestral Wuhan Hu-1 (n = 21), and 13 others (n = 26). Public health intervention may have curtailed the persistence of lineages that appeared transiently during February and March. By August, only D614G lineages introduced after 21 March were circulating in Santa Clara County.

Quantitative diffusion MRI (dMRI) is a promising technique for evaluating the spinal cord in health and disease. However, low signal-to-noise ratio (SNR) can impede interpretation and quantification of these images. The purpose of this study is to evaluate a denoising approach, Patch2Self, to improve the quality, reliability, and accuracy of quantitative diffusion MRI of the spinal cord. Patch2Self is a self-supervised learning-based denoising method that leverages statistical independence of noise to suppress signal components strictly originating from random fluctuations. We conduct three experiments to validate the denoising performance of Patch2Self on clinical-quality, single-shell dMRI acquisitions with a small number of gradient directions: 1) inter-session scanrescan in healthy volunteers to evaluate enhancements in image contrast and model fitting; 2) repeated intra-session scans in a healthy volunteer to compare signal averaging to Patch2Self; and 3) assessment of spinal cord lesion conspicuity in a multiple sclerosis group. We find that Patch2Self improves intra-cord contrast, signal modeling, SNR, and lesion conspicuity within the spinal cord. This denoising approach holds promise for facilitating reliable diffusion measurements in the spinal cord to investigate biological and pathological processes.

Mosquitoes are a disease vector with a complex ecology involving interactions between transmissible pathogens, endogenous microbiota, and human and animal blood meal sources. Unbiased metatranscriptomic sequencing of individual mosquitoes offers a straightforward and rapid way to characterize these dynamics. Here, we profile 148 diverse wild-caught mosquitoes collected in California, detecting sequences from eukaryotes, prokaryotes, and over 70 known and novel viral species. Because we sequenced singletons, it was possible to compute the prevalence of each microbe and recognize a high frequency of viral co-infection. By analyzing the pattern of co-occurrence of sequences across samples, we associated “dark matter” sequences with recognizable viral polymerases, and animal pathogens with specific blood meal sources. We were also able to detect frequent genetic reassortment events in a highly prevalent quaranjavirus undergoing a recent intercontinental sweep. In the context of an emerging disease, where knowledge about vectors, pathogens, and reservoirs is lacking, the approaches described here can provide actionable information for public health surveillance and intervention decisions.

We propose a general framework for solving inverse problems in the presence of noise that requires no signal prior, no noise estimate, and no clean training data. We only require that the forward model be available and that the noise be statistically independent across measurement dimensions. We build upon the theory of 𝒥-invariant functions (Batson Royer 2019, arXiv:1901.11365) and show how self-supervised denoising à la Noise2Self is a special case of learning a noise-tolerant pseudo-inverse of the identity. We demonstrate our approach by showing how a convolutional neural network can be taught in a self-supervised manner to deconvolve images and surpass in image quality classical inversion schemes such as Lucy-Richardson deconvolution.