Answering questions over multi-page, multimodal documents, including text and figures, is a critical challenge for applications that require answers to integrate information across multiple modalities and contextual dependencies. Existing methods, such as single-turn retrieval-augmented generation (RAG), struggle to retrieve fine-grained and contextually relevant information from large, heterogeneous documents, leading to suboptimal performance. Inspired by iterative frameworks like ReAct, which refine retrieval through feedback, we propose Doc-React, an adaptive iterative framework that balances information gain and uncertainty reduction at each step. Doc-React leverages InfoNCE-guided retrieval to approximate mutual information, enabling dynamic sub-query generation and refinement. A large language model (LLM) serves as both a judge and generator, providing structured feedback to iteratively improve retrieval. By combining mutual information optimization with entropy-aware selection, Doc-React systematically captures relevant multimodal content, achieving strong performance on complex QA tasks

Enterprise AI Assistants are increasingly deployed in domains where accuracy is paramount, making each erroneous output a potentially significant incident. This paper presents a comprehensive framework for monitoring, benchmarking, and continuously improving such complex, multi-component systems under active development by multiple teams. Our approach encompasses three key elements: (1) a hierarchical “severity” framework for incident detection that identifies and categorizes errors while attributing component-specific error rates, facilitating targeted improvements; (2) a scalable and principled methodology for benchmark construction, evaluation, and deployment, designed to accommodate multiple development teams, mitigate overfitting risks, and assess the downstream impact of system modifications; and (3) a continual improvement strategy leveraging multidimensional evaluation, enabling the identification and implementation of diverse enhancement opportunities. By adopting this holistic framework, organizations can systematically enhance the reliability and performance of their AI Assistants, ensuring their efficacy in critical enterprise environments. We conclude by discussing how this multifaceted evaluation approach opens avenues for various classes of enhancements, paving the way for more robust and trustworthy AI systems.

Large Language Models (LLMs) are increasingly integrated into diverse applications. The rapid evolution of LLMs presents opportunities for developers to enhance applications continuously. However, this constant adaptation can also lead to performance regressions during model migrations. While several interactive tools have been proposed to streamline the complexity of prompt engineering, few address the specific requirements of regression testing for LLM Migrations. To bridge this gap, we introduce RETAIN (REgression Testing guided LLM migrAtIoN), a tool designed explicitly for regression testing in LLM Migrations. RETAIN comprises two key components: an interactive interface tailored to regression testing needs during LLM migrations, and an error discovery module that facilitates understanding of differences in model behaviors. The error discovery module generates textual descriptions of various errors or differences between model outputs, providing actionable insights for prompt refinement. Our automatic evaluation and empirical user studies demonstrate that RETAIN, when compared to manual evaluation, enabled participants to identify twice as many errors, facilitated experimentation with 75% more prompts, and achieves 12% higher metric scores in a given time frame.

The development of conversational AI assistants is an iterative process with multiple components. As such, the evaluation and continual improvement of these assistants is a complex and multifaceted problem. This paper introduces the challenges in evaluating and improving a generative AI assistant for enterprises, which is under active development, and how we address these challenges. We also share preliminary results and discuss lessons learned.

A/B tests are the gold standard for evaluating digital experiences on the web. However, traditional "fixed-horizon" statistical methods are often incompatible with the needs of modern industry practitioners as they do not permit continuous monitoring of experiments. Frequent evaluation of fixed-horizon tests ("peeking") leads to inflated type-I error and can result in erroneous conclusions. We have released an experimentation service on the Adobe Experience Platform based on anytime-valid confidence sequences, allowing for continuous monitoring of the A/B test and data-dependent stopping. We demonstrate how we adapted and deployed asymptotic confidence sequences in a full featured A/B testing platform, describe how sample size calculations can be performed, and how alternate test statistics like "lift" can be analyzed. On both simulated data and thousands of real experiments, we show the desirable properties of using anytime-valid methods instead of traditional approaches.

Machine learning-based predictions are popular in many applications including healthcare, recommender systems and finance. More recently, the development of low-end edge hardware (e.g., Apple’s Neural Engine and Intel’s Movidius VPU) has provided a path for the proliferation of machine learning on the edge with on-device modeling. Modeling on the device reduces latency and helps maintain the user’s privacy. However, on-device modeling can leak private server-side information. In this work, we investigate on-device machine learning models that are used to provide a service and propose novel privacy attacks that can leak sensitive proprietary information of the service provider. We demonstrate that different adversaries can easily exploit such models to maximize their profit and accomplish content theft. Motivated by the need to preserve both client and server privacy, we present preliminary ideas on thwarting such attacks.

Nematic order has manifested itself in a variety of materials in the cuprate family. We propose an effective field theory of a layered system with incommensurate, intertwined spin- and charge-density wave (SDW and CDW) orders, each of which consists of two components related by ${C}_{4}$ rotations. Using a variational method (which is exact in a large-$N$ limit), we study the development of nematicity from partially melting those density waves by either increasing temperature or adding quenched disorder. As temperature decreases we first find a transition to a single nematic phase, but depending on the range of parameters (e.g., doping concentration) the strongest fluctuations associated with this phase reflect either proximate SDW or CDW order. We also discuss the changes in parameters that can account for the differences in the SDW-CDW interplay between the 214 family and the other hole-doped cuprates.

Received 19 September 2017DOI:https://doi.org/10.1103/PhysRevB.96.119911©2017 American Physical SocietyPhysics Subject Headings (PhySH)Research AreasElectrical conductivityFermi surfaceHall effectMetalsQuantum phase transitionsSuperconductivityCondensed Matter, Materials & Applied Physics

In the context of the relaxation time approximation to Boltzmann transport theory, we examine the behavior of the Hall number n(H) of a metal in the neighborhood of a Lifshitz transition from a closed Fermi surface to open sheets. We find a universal nonanalytic dependence of n(H) on the electron density in the high-field limit, but a nonsingular dependence at low fields. The existence of an assumed nematic transition produces a doping dependent n(H) similar to that observed in recent experiments in the high-temperature superconductor YBa2Cu3O7-x.

The paradigmatic example of a continuous quantum phase transition is the transverse field Ising ferromagnet. In contrast to classical critical systems, whose properties depend only on symmetry and the dimension of space, the nature of a quantum phase transition also depends on the dynamics. In the transverse field Ising model, the order parameter is not conserved, and increasing the transverse field enhances quantum fluctuations until they become strong enough to restore the symmetry of the ground state. Ising pseudospins can represent the order parameter of any system with a twofold degenerate broken-symmetry phase, including electronic nematic order associated with spontaneous point-group symmetry breaking. Here, we show for the representative example of orbital-nematic ordering of a non-Kramers doublet that an orthogonal strain or a perpendicular magnetic field plays the role of the transverse field, thereby providing a practical route for tuning appropriate materials to a quantum critical point. While the transverse fields are conjugate to seemingly unrelated order parameters, their nontrivial commutation relations with the nematic order parameter, which can be represented by a Berry-phase term in an effective field theory, intrinsically intertwine the different order parameters.

We show that the adversarial robustness of deep neural networks can be improved by reducing the dimensionality of input data. Adversarial examples are generated by small perturbations of regular input data, and while these changes are typically imperceptible to the human eye, they can result in catastrophic losses of accuracy in neural networks. This has been shown to be a generic property of quasi-linear classifiers in high dimensional spaces; thus, it is intuitively possible that reducing the dimensionality of the inputs leaves less ‘space’ for adversarial examples to exist. We present detailed results demonstrating this improved robustness (with small sacrifices in accuracy) for 5 layer networks on the CIFAR10 dataset, as well as for the deeper CaffeNet and GoogLeNet architectures on the Tiny-Imagenet-200 dataset. In particular, a reduction in the number of allowed pixel intensities from 256 to 4, which corresponds to robustness to perturbations of up to 32 units of intensity, results in an accuracy reduction of only 6% for CIFAR10 and 8% for Tiny-Imagenet.

The existence of charge density wave (CDW) correlations in cuprate superconductors has now been established. However, the nature of the ground state order has remained uncertain because disorder and the presence of superconductivity typically limit the CDW correlation lengths to a dozen unit cells or less. Here we explore the CDW correlations in YBa2Cu3Ox (YBCO) ortho-II and ortho-VIII crystals, which belong to the cleanest available cuprate family, at magnetic fields in excess of the resistive upper critical field (Hc2) where the superconductivity is heavily suppressed. We find an incommensurate, unidirectional CDW with a well-defined onset at a critical field strength that is proportional to Hc2. It is related to but distinct from the short-range bidirectional CDW that exists at zero magnetic field. The unidirectional CDW possesses a long inplane correlation length as well as significant correlations between neighboring CuO2 planes, yielding a correlation volume that is at least 2 - 3 orders of magnitude larger than that of the zero-field CDW. This is by far the largest CDW correlation volume observed in any cuprate crystal and so is presumably representative of the high-field ground-state of an "ideal" disorder-free cuprate.

Using an exact numerical solution and semiclassical analysis, we investigate quantum oscillations (QOs) in a model of a bilayer system with an anisotropic (elliptical) electron pocket in each plane. Key features of QO experiments in the high temperature superconducting cuprate YBCO can be reproduced by such a model, in particular the pattern of oscillation frequencies (which reflect "magnetic breakdown" between the two pockets) and the polar and azimuthal angular dependence of the oscillation amplitudes. However, the requisite magnetic breakdown is possible only under the assumption that the horizontal mirror plane symmetry is spontaneously broken and that the bilayer tunneling t(perpendicular to) is substantially renormalized from its 'bare' value. Under the assumption that t(perpendicular to) = (Z) over tildet(perpendicular to)((0)), where (Z) over tilde is a measure of the quasiparticle weight, this suggests that (Z) over tilde less than or similar to 1/20. Detailed comparisons with new YBa2Cu3O6.58 QO data, taken over a very broad range of magnetic field, confirm specific predictions made by the breakdown scenario.

The elastoresistivity tensor m(ij,kl) characterizes changes in a material's resistivity due to strain. As a fourth-rank tensor, elastoresistivity can be a uniquely useful probe of the symmetries and character of the electronic state of a solid. We present a symmetry analysis of mij, kl (both in the presence and absence of a magnetic field) based on the crystalline point group, focusing for pedagogic purposes on the D-4h point group (of relevance to several materials of current interest). We also discuss the relation between m(ij,kl) and various thermodynamic susceptibilities, particularly where they are sensitive to critical fluctuations proximate to a critical point at which a point-group symmetry is spontaneously broken.

Harper's equation (aka the "almost Mathieu" equation) famously describes the quantum dynamics of an electron on a one dimensional lattice in the presence of an incommensurate potential with magnitude V and wave number Q. It has been proven that all states are delocalized if V is less than a critical value V_c=2t and localized if V> V_c. Here, we show that this result (while correct) is highly misleading, at least in the small Q limit. In particular, for VE_c, form a set of narrow bands with exponentially small bandwidths ∼ t exp[-(2πα/Q)] (where α is an energy dependent number of order 1) separated by band-gaps ∼ t Q. Thus, the states with |E|> E_c are "almost localized" in that they have an exponentially large effective mass and are easily localized by small perturbations. We establish this both using exact numerical solution of the problem, and by exploiting the well known fact that the same eigenvalue problem arises in the Hofstadter problem of an electron moving on a 2D lattice in the presence of a magnetic field, B=Q/2π. From the 2D perspective, the almost localized states are simply the Landau levels associated with semiclassical precession around closed contours of constant quasiparticle energy; that they are not truly localized reflects an extremely subtle form of magnetic breakdown.

For materials that harbour a continuous phase transition, the susceptibility of the material to various fields can be used to understand the nature of the fluctuating order and hence the nature of the ordered state. Here we use anisotropic biaxial strain to probe the nematic susceptibility of URu2Si2, a heavy fermion material for which the nature of the low temperature 'hidden order' state has defied comprehensive understanding for over 30 years. Our measurements reveal that the fluctuating order has a nematic component, confirming reports of twofold anisotropy in the broken symmetry state and strongly constraining theoretical models of the hidden-order phase.

Because a material with an incommensurate charge density wave (ICDW) is only quasi-periodic, Bloch's theorem does not apply and there is no sharply defined Fermi surface. We will show that, as a consequence, there are no quantum oscillations which are truly periodic functions of $1/B$ (where $ B$ is the magnitude of an applied magnetic field). For a weak ICDW, there exist broad ranges of $1/B$ in which approximately periodic variations occur, but with frequencies that vary inexorably in an unending cascade with increasing $1/B$. For a strong ICDW, e.g. in a quasi-crystal, no quantum oscillations survive at all. Rational and irrational numbers really are different.

We propose the possible detection of broken mirror symmetries in correlated two-dimensional materials by elastotransport measurements. Using linear response theory we calculate the "shear conductivity" Gamma(xx,) (xy), defined as the linear change of the longitudinal conductivity sigma(xx) due to a shear strain c(xy). This quantity can only be nonvanishing when in-plane mirror symmetries are broken and we discuss how candidate states in the cuprate pseudogap regime (e.g., various loop current or charge orders) may exhibit a finite shear conductivity. We also provide a realistic experimental protocol for detecting such a response.