What does a language model predict when it has few clues? The answer lurks in its unembedding geometry: a single direction of the unembedding matrix encodes the unigram distribution of the training corpus, which serves as the Bayesian prior the model falls back on when uncertain. This structure — which we term the direction of ignorance — appears in all four model families examined (, , , and ), ranging from 0.4B to 405B parameters. Projecting the final prediction state onto this direction yields a per-token prior loading factor λ, which, empirically, declines steadily as the context becomes more informative. Formally, the same projection decomposes the prediction state into two orthogonal vectors that correspond exactly to the two factors of a tempered Bayesian update: a unigram prior raised to the exponent λ and a context-driven likelihood. This geometric-probabilistic interpretation calibrates λ, making it meaningfully comparable across model sizes and families, with larger models generally exhibiting lower prior reliance in the high-context limit. Finally, we show that the direction of ignorance is causally active: raising or lowering λ at the final prediction state steers the prediction toward or away from the unigram prior in KL divergence.

Pretrained large language models (LLMs) have demonstrated in-context learning (ICL) capabilities for numerical inference over sequences serialized as text. Prior work has identified and characterized this form of numerical inference primarily through output-level evaluations such as prediction error. However, how numerical information is organized within LLM representations remains much less understood. To study this internal organization, we adopt a graph signal processing perspective in which attention induces a weighted graph over tokens, while token hidden states define signals on its nodes. Quantitative graph-spectral diagnostics and qualitative token-graph visualizations reveal that representations become more clearly differentiated by input dynamical complexity as context length increases. Simpler inputs produce attention-induced token graphs with stronger global connectivity and smoother, spectrally concentrated hidden-state signals, whereas more complex inputs produce more localized graphs and hidden-state signals with broader spectral support and greater high-frequency energy. Together, these findings point to systematic, context-dependent internal signatures associated with numerical ICL that are conserved across model families.

Large language models fine-tuned for forecasting can be accurate yet poorly calibrated, and their chain-of-thought (CoT) reasoning may not faithfully reflect the evidence behind a forecast. We ask whether internal representations offer a more direct window into both. Working with Eternis-Forecaster 8B on OpenForesight, we train representation-pooling probes on intermediate activations and find they achieve substantially better calibration; a result that also holds for GLM-4.7-Flash and GLM-4.5-Air. We then assess CoT faithfulness through evidence ablation and diversionary injection: removing an influential source in the prompt often changes the model's forecast while leaving the reasoning trace untouched. The same probes function as lie detectors: their activations track behavioral shifts far better than the reasoning trace does, and they also predict the direction of change in 84

Large language models (LLMs) make next-token predictions based on clues present in their context, such as semantic descriptions and in-context examples. Yet, elucidating which prior tokens most strongly influence a given prediction remains challenging due to the proliferation of layers and attention heads in modern architectures. We propose Jacobian Scopes, a suite of gradient-based, token-level causal attribution methods for interpreting LLM predictions. Grounded in perturbation theory and information geometry, Jacobian Scopes quantify how input tokens influence various aspects of a model's prediction, such as specific logits, the full predictive distribution, and model uncertainty (effective temperature). Through case studies spanning instruction understanding, translation, and in-context learning (ICL), we demonstrate how Jacobian Scopes reveal implicit political biases, uncover word- and phrase-level translation strategies, and shed light on recently debated mechanisms underlying in-context time-series forecasting. To facilitate exploration of Jacobian Scopes on custom text, we open-source our implementations and provide a cloud-hosted interactive demo at https://huggingface.co/spaces/Typony/JacobianScopes.

Large language models (LLMs) demonstrate remarkable emergent abilities to perform in-context learning across various tasks, including time series forecasting. This work investigates LLMs' ability to estimate probability density functions (PDFs) from data observed in-context; such density estimation (DE) is a fundamental task underlying many probabilistic modeling problems. We leverage the Intensive Principal Component Analysis (InPCA) to visualize and analyze the in-context learning dynamics of LLaMA-2 models. Our main finding is that these LLMs all follow similar learning trajectories in a low-dimensional InPCA space, which are distinct from those of traditional density estimation methods like histograms and Gaussian kernel density estimation (KDE). We interpret the LLaMA in-context DE process as a KDE with an adaptive kernel width and shape. This custom kernel model captures a significant portion of LLaMA's behavior despite having only two parameters. We further speculate on why LLaMA's kernel width and shape differs from classical algorithms, providing insights into the mechanism of in-context probabilistic reasoning in LLMs.Our codebase, along with a 3D visualization of an LLM's in-context learning trajectory, is publicly available at https://github.com/AntonioLiu97/LLMICL_inPCA.

Large language models (LLMs) have demonstrated emergent in-context learning (ICL) capabilities across a range of tasks, including zero-shot time-series forecasting. We show that text-trained foundation models can accurately extrapolate spatiotemporal dynamics from discretized partial differential equation (PDE) solutions without fine-tuning or natural language prompting. Predictive accuracy improves with longer temporal contexts but degrades at finer spatial discretizations. In multi-step rollouts, where the model recursively predicts future spatial states over multiple time steps, errors grow algebraically with the time horizon, reminiscent of global error accumulation in classical finite-difference solvers. We interpret these trends as in-context neural scaling laws, where prediction quality varies predictably with both context length and output length. To better understand how LLMs are able to internally process PDE solutions so as to accurately roll them out, we analyze token-level output distributions and uncover a consistent three-stage ICL progression: beginning with syntactic pattern imitation, transitioning through an exploratory high-entropy phase, and culminating in confident, numerically grounded predictions.

In this work, we present a mesh-independent, data-driven library, chebgreen, to mathematically model onedimensional systems, possessing an associated control parameter, and whose governing partial differential equation is unknown. The proposed method learns an Empirical Green's Function for the associated, but hidden, boundary value problem, in the form of a Rational Neural Network from which we subsequently construct a bivariate representation in a Chebyshev basis. We uncover the Green's function, at an unseen control parameter value, by interpolating the left and right singular functions within a suitable library, expressed as points on a manifold of Quasimatrices, while the associated singular values are interpolated with Lagrange polynomials. This work improves upon prior work by extending the scope of applicability to non-self-adjoint operators and improves data efficiency.

Large Language Models achieve next-token prediction by transporting a vectorized piece of text (prompt) across an accompanying embedding space under the action of successive transformer layers. The resulting high-dimensional trajectories realize different contextualization, or 'thinking', steps, and fully determine the output probability distribution. We aim to characterize the statistical properties of ensembles of these 'lines of thought.' We observe that independent trajectories cluster along a low-dimensional, non-Euclidean manifold, and that their path can be well approximated by a stochastic equation with few parameters extracted from data. We find it remarkable that the vast complexity of such large models can be reduced to a much simpler form, and we reflect on implications.

In this paper, we introduce PDE-LEARN, a novel deep learning algorithm that can identify governing partial differential equations (PDEs) directly from noisy, limited measurements of a physical system of interest. PDE-LEARN uses a Rational Neural Network, U, to approximate the system response function and a sparse, trainable vector, ξ, to characterize the hidden PDE that the system response function satisfies. Our approach couples the training of U and ξ using a loss function that (1) makes U approximate the system response function, (2) encapsulates the fact that U satisfies a hidden PDE that ξ characterizes, and (3) promotes sparsity in ξ using ideas from iteratively reweighted least-squares. Further, PDE-LEARN can simultaneously learn from several data sets, allowing it to incorporate results from multiple experiments. This approach yields a robust algorithm to discover PDEs directly from realistic scientific data. We demonstrate the efficacy of PDE-LEARN by identifying several PDEs from noisy and limited measurements.

We introduce Weak-PDE-LEARN, a Partial Differential Equation (PDE) discovery algorithm that can identify non-linear PDEs from noisy, limited measurements of their solutions. Weak-PDE-LEARN uses an adaptive loss function based on weak forms to train a neural network, $U$, to approximate the PDE solution while simultaneously identifying the governing PDE. This approach yields an algorithm that is robust to noise and can discover a range of PDEs directly from noisy, limited measurements of their solutions. We demonstrate the efficacy of Weak-PDE-LEARN by learning several benchmark PDEs.

Pretrained large language models (LLMs) are surprisingly effective at performing zero-shot tasks, including time-series forecasting. However, understanding the mechanisms behind such capabilities remains highly challenging due to the complexity of the models. We study LLMs' ability to extrapolate the behavior of dynamical systems whose evolution is governed by principles of physical interest. Our results show that LLaMA 2, a language model trained primarily on texts, achieves accurate predictions of dynamical system time series without fine-tuning or prompt engineering. Moreover, the accuracy of the learned physical rules increases with the length of the input context window, revealing an in-context version of neural scaling law. Along the way, we present a flexible and efficient algorithm for extracting probability density functions of multi-digit numbers directly from LLMs.

We present a data-driven approach to mathematically model physical systems whose governing partial differential equations are unknown, by learning their associated Green's function. The subject systems are observed by collecting input-output pairs of system responses under excitations drawn from a Gaussian process. Two methods are proposed to learn the Green's function. In the first method, we use the proper orthogonal decomposition (POD) modes of the system as a surrogate for the eigenvectors of the Green's function, and subsequently fit the eigenvalues, using data. In the second, we employ a generalization of the randomized singular value decomposition (SVD) to operators, in order to construct a low-rank approximation to the Green's function. Then, we propose a manifold interpolation scheme, for use in an offline-online setting, where offline excitation-response data, taken at specific model parameter instances, are compressed into empirical eigenmodes. These eigenmodes are subsequently used within a manifold interpolation scheme, to uncover other suitable eigenmodes at unseen model parameters. The approximation and interpolation numerical techniques are demonstrated on several examples in one and two dimensions.

Scientific machine learning has been successfully applied to inverse problems and PDE discovery in computational physics. One caveat concerning current methods is the need for large amounts of ("clean") data, in order to characterize the full system response and discover underlying physical models. Bayesian methods may be particularly promising for overcoming these challenges, as they are naturally less sensitive to the negative effects of sparse and noisy data. In this paper, we propose to use Bayesian neural networks (BNN) in order to: 1) Recover the full system states from measurement data (e.g. temperature, velocity field, etc.). We use Hamiltonian Monte-Carlo to sample the posterior distribution of a deep and dense BNN, and show that it is possible to accurately capture physics of varying complexity, without overfitting. 2) Recover the parameters instantiating the underlying partial differential equation (PDE) governing the physical system. Using the trained BNN, as a surrogate of the system response, we generate datasets of derivatives that are potentially comprising the latent PDE governing the observed system and then perform a sequential threshold Bayesian linear regression (STBLR), between the successive derivatives in space and time, to recover the original PDE parameters. We take advantage of the confidence intervals within the BNN outputs, and introduce the spatial derivatives cumulative variance into the STBLR likelihood, to mitigate the influence of highly uncertain derivative data points; thus allowing for more accurate parameter discovery. We demonstrate our approach on a handful of example, in applied physics and non-linear dynamics.

There is an opportunity for deep learning to revolutionize science and technology by revealing its findings in a human interpretable manner. To do this, we develop a novel data-driven approach for creating a human-machine partnership to accelerate scientific discovery. By collecting physical system responses under excitations drawn from a Gaussian process, we train rational neural networks to learn Green's functions of hidden linear partial differential equations. These functions reveal human-understandable properties and features, such as linear conservation laws and symmetries, along with shock and singularity locations, boundary effects, and dominant modes. We illustrate the technique on several examples and capture a range of physics, including advection-diffusion, viscous shocks, and Stokes flow in a lid-driven cavity.

PDE discovery shows promise for uncovering predictive models of complex physical systems but has difficulty when measurements are noisy and limited. We introduce a new approach for PDE discovery that uses two Rational Neural Networks and a principled sparse regression algorithm to identify the hidden dynamics that govern a system’s response. The first network learns the system response function, while the second learns a hidden PDE describing the system’s evolution. We then use a parameter-free sparse regression algorithm to extract a human-readable form of the hidden PDE from the second network. We implement our approach in an open-source library called PDE-READ. Our approach successfully identifies the governing PDE in six benchmark examples. We demonstrate that our approach is robust to both sparsity and noise and it, therefore, holds promise for application to real-world observational data.

Consideration of slamming loads within the structural design of planning hulls is of critical importance in ensuring adequate structural performance. However, a general engineering theory for slamming has yet to be uncovered due to the intricacy in the interplay between complex fluid flows and nonlinear structural deformations, and so design relies on specialized theories. We propose one such theory for a design case that has, until now, eluded a proper description. In pursuit of this theory, we employ an implicit, partitioned fluid-structural interaction (FSI) simulation approach, to study the underlying physical mechanisms accompanying the oblique impact of a flexible plate during water entry. In the present work, we first present validation results from flexible plate water entry experiments (our FSI results display satisfactory alignment compared with experimental data). Subsequently, we carry out a series of numerical analyses to characterize the impact force and plate out-of-plane deformations. Finally, we use our FSI solver to study the mechanistic evolution of fluid flows and elastic plate deformations that occur during slamming. Based on these observations, we propose a novel engineering theory for flexible plates obliquely impacting the water free surface. Our proposed theory presents excellent predicative capability compared with FSI solutions.

Certifying performance of mechanical components with experimental tests is time consuming and expensive, which motivates the development of efficient approaches for predicting the outcomes from such testing. We propose two methods based on Gaussian processes (GP) to estimate the probability that new components will pass future certification tests, while assessing our prediction confidence. The first method processes a set of Bernoulli trials into a suitable machine learning dataset and subsequently infers the probability of performing satisfactorily for new components using heteroscedastic bounded GP regression. The second method uses GP classification with linear kernels. We demonstrate that linear kernels are well suited for datasets representing snapshots of mechanical system responses by accurately reproducing the underlying physical trends in the data. This yields consistent probabilities of passing and provides high labeling accuracy, even with small datasets. We demonstrate these techniques on synthetic datasets consistent with ship cabinet certification tests. We achieve up to 100% accuracy using all of the training data, and at least 92% with only 10% of the available data. With a corrupted training set, we obtain at least 93% accuracy. In the regression framework, we demonstrate that introducing heteroscedasticity helps achieve significantly better accuracy than frequentist machine learning methods.

Real-time characterization of refractivity within the marine atmospheric boundary layer can provide valuable information that can potentially be used to mitigate the effects of atmospheric ducting on radar performance. Many duct characterization models are successful at predicting parameters from a specific refractivity profile associated with a given type of duct; however, the ability to classify, and then subsequently characterize, various duct types is an important step towards a more comprehensive prediction model. We introduce a two-step approach using deep learning to differentiate sparsely sampled propagation factor measurements collected under evaporation ducting conditions with those collected under surface-based ducting conditions in order to subsequently estimate the appropriate refractivity parameters based on that differentiation. We show that this approach is not only accurate, but also efficient; thus providing a suitable method for real-time applications.

In applied science and engineering contexts, ocean free-surface wave-field characterization is an ongoing research area relevant to many diverse applications. The use of optical means for this characterization, in particular, is an appealing solution due to the promise of relatively high spatial resolution and modest cost (i.e., applying off-the-shelf still cameras or video cameras). This work investigates the validity of a previously published algorithm related to visible-wavelength ocean free-surface wave-field videos. The first part of the algorithm uses the video content to determine scale from a single-perspective, uncalibrated video. The second part of the algorithm applies the video content to determine sea-state characteristics, such as dominant wave period and wavelength. Due to the complications of obtaining well-characterized real-world data sets, algorithms proposed within the literature are not always validated experimentally, but the scientific community relies on the process of verification and validation to build confidence in emerging scientific contributions. This work addresses the need to assess the performance of the existing algorithm through validation experiments. Video recordings of approximate deep-water waves are collected in a controlled indoor environment to experimentally validate the algorithm, and a suite of postprocessing parameters is explored to identify the algorithm's sensitivities for a chosen unit problem. Recommendations for the algorithm parameter selection and an assessment of the algorithm efficacy are offered for researchers interested in the implementation of this algorithm.

In this work, we consider a thermographic inspection setup for detection and characterization of corrosion in a steel structure. Solid products of corrosion, if present, interact with the flow of heat within a domain, so that a structure's thermal response to a known energy input can provide information about internal damage. We focus our study on pitting corrosion, which complicates the detection and characterization tasks because the damage can be both small and hidden on inaccessible areas of the structure. Nevertheless, pitting corrosion poses a threat to structural components, so early detection is important. We develop a procedure based on Bayesian mathematical analysis to achieve rigorous inference over locations and geometric forms of hidden corrosion pits based on their subtle influences within noisy temperature measurements made on observable portions of the structure. A strategy is developed for optimizing a thermographic framework for the purpose of maximizing this response. Numerical experiments are performed to demonstrate our proposed corrosion detection and characterization procedure. It is found that reliable inference can be done on corrosion pits millimeters deep in the rear side of a steel panel using observations on the front side only. The Bayesian inference method is also shown to be effective in more complex structural geometries. (C) 2020 Elsevier Ltd. All rights reserved.