美国运通

Evaluating user-facing AI applications remains a central challenge, especially in open-ended domains such as travel planning, clinical note generation, or dialogue. The gold standard is user feedback (e.g., thumbs up/down) or behavioral signals (e.g., retention), but these are often scarce in prototypes and research projects, or too-slow to use for system optimization. We present **AutoMetrics**, a framework for synthesizing evaluation metrics under low-data constraints. AutoMetrics combines retrieval from **MetricBank**, a collection of 48 metrics we curate, with automatically generated LLM-as-a-Judge criteria informed by lightweight human feedback. These metrics are composed via regression to maximize correlation with human signal. AutoMetrics takes you from expensive measures to interpretable automatic metrics. Across 5 diverse tasks, AutoMetrics improves Kendall correlation with human ratings by up to 33.4% over LLM-as-a-Judge while requiring fewer than 100 feedback points. We show that AutoMetrics can be used as a proxy reward to equal effect as a verifiable reward. We release the full AutoMetrics toolkit and MetricBank to accelerate adaptive evaluation of LLM applications.

Large language models (LLMs) are becoming useful in many domains due to their impressive abilities that arise from large training datasets and large model sizes. However, research on LLM-based approaches to document inconsistency detection is relatively limited. We address this gap by investigating evidence extraction capabilties of LLMs for document inconsistency detection. To this end, we introduce new comprehensive evidence-extraction metrics and a redact-and-retry framework with constrained filtering that substantially improves evidence extraction performance over other prompting methods. We support our approach with strong experimental results and release a new semi-synthetic dataset for evaluating evidence extraction.

Federated Learning (FL) trains a global model across decentralized clients while preserving data privacy, but at scale it is vulnerable to malicious updates. Byzantine-resilient aggregation methods such as MultiKrum score gradients against their nearest neighbors and can miss malicious updates that preserve the statistical properties of honest ones. We propose a quantum annealing approach that reformulates client selection as a Quadratic Unconstrained Binary Optimization (QUBO) problem, encoding pairwise distances into a cost function solved by quantum annealers (QA). Unlike MultiKrum's greedy per-client scoring, the QUBO formulation jointly optimizes over all subsets to find the mutually closest group of m clients. At small scale (15 clients), QUBO outperforms MultiKrum on the most challenging Byzantine attacks: e.g., Advanced LIE is detected with 95.11

The modeling of credit risk within the consortium based financial settings brings about great complexity, owing to the distributed ownership of data and the strict privacy considerations. Traditional risk assessment methods are usually based on a centralized data processing, and this means that sensitive financial data is at risk of misuse, there is a question of data integrity and there is lack of transparent audit trails. The recent development of privacy-preserving computation approaches like homomorphic encryption (HE) creates new possibilities as they allow performing mathematical operations on encrypted credit data without disclosing unencrypted values. Nonetheless, although HE maintains privacy, it does not intrinsically imply transparency, and thus it is hard to tell whether the intermediate computations are accurate, particularly in the multi-institutional scenario. In order to overcome these shortcomings, the block chain technology is proposed as an additional audit layer. The decentralized ledger design of block chain makes the computation log immutable and traceable, so that the participants of a consortium can verify the result independently without revealing the origin data. The combination of HE with block chain provides a secure framework of federated credit risk modelling, which concurrently protects privacy, enhanced computational verifiability and transparent decision-making. In this paper, we suggest a hybrid system, which integrates HE-based secure computation with block chain-verified audit trail in the credit risk assessment in consortium networks. The performance of the framework is assessed on a simulated dataset in comparison to standard HE-only methods. In both partitioning’s risk scores are calculated and the models are evaluated against standard performance measures, such as average prediction variance, computational efficiency, auditability and robustness to stochastic input perturbations. Findings show that the block chain-audited HE model is effective in improving robustness and accountability of the credit evaluations. The suggested framework is a decent solution to cooperative financial ecosystems where privacy, accuracy, and traceability are essential. The results lead to the future of secure and auditable financial modelling in decentralized and privacy-preserving settings.

This study delves into applying blockchain technology to maximize trust and efficiency when it comes to the process of verifying insurance claims, in health and motor car insurance markets in this case. Through the application of a permissioned blockchain network, the study proves how blockchain transparency and decentralization can facilitate processing claims yet maintain data integrity and minimize fraud. Four core algorithms—Blockchain Consensus, Smart Contracts, Fraud Detection, and Claim Validation—were developed and implemented. The tests registered a reduction of 30