Evaluating the quality of open-domain chatbots has become increasingly reliant on LLMs acting as automatic judges. However, existing meta-evaluation benchmarks are static, outdated, and lacking in multilingual coverage, limiting their ability to fully capture subtle weaknesses in evaluation. We introduce MEDAL, an automated multi-agent framework for curating more representative and diverse open-domain dialogue evaluation benchmarks. Our approach leverages several state-of-the-art LLMs to generate user-chatbot multilingual dialogues, conditioned on varied seed contexts. Then, a strong LLM (GPT-4.1) is used for a multidimensional analysis of the performance of the chatbots, uncovering noticeable cross-lingual performance differences. Guided by this large-scale evaluation, we curate a new meta-evaluation multilingual benchmark and human-annotate samples with nuanced quality judgments. This benchmark is then used to assess the ability of several reasoning and non-reasoning LLMs to act as evaluators of open-domain dialogues. Using MEDAL, we uncover that state-of-the-art judges fail to reliably detect nuanced issues such as lack of empathy, commonsense, or relevance.
Large language models (LLMs) have demonstrated remarkable capabilities in analyzing textual data to support disease diagnosis. Extending these advances to audio, Multimodal Large Language Models (MLLMs) open new opportunities for addressing speech-impairing conditions such as Parkinson’s Disease (PD). Using the NeuroVoz corpus, which includes speech recordings and expert evaluations across 14 perceptual dimensions of voice quality, phonation, and prosody, we validated the clinical utility of these annotations and assessed the ability of MLLMs to replicate them. The models generated acoustic macro-descriptors that showed good agreement with expert ratings and enabled effective PD classification. Feeding these descriptors into machine learning classifiers for PD detection achieved up to 80.47% UAR. Overall, the findings highlight the potential of MLLMs to provide reliable and interpretable features directly from audio, thereby supporting scalable and cross-domain speech-based diagnosis of neurodegenerative conditions.
OBJECTIVE:Forensic dental age assessment is required when documentary evidence is absent or unreliable, and judicial decisions often depend on whether an individual falls below or above legally defined age thresholds. This pilot study evaluates ImageNet pretrained convolutional neural network architectures for binary age threshold classification. DESIGN:Orthopantomograms from individuals aged 0-25 years (1887 males and 1664 females) was selected using predefined inclusion criteria and labelled by chronological age. For each legal threshold (10, 12, 14, 16, 18 and 21 years), images were stratified into training and validation sets using an 80 20 split while preserving class distributions. Preprocessing included contrast enhancement with contrast limited adaptive histogram equalization, automated cropping, resizing to model specific input dimensions and ImageNet normalisation. Data augmentation was applied only to training images. Seven convolutional neural network architectures (ResNet 50, ResNet 152, VGG19, DenseNet 121, DenseNet 169, EfficientNetV2 M and Xception) were fine tuned in PyTorch using binary cross entropy loss with early stopping. Hyperparameters were optimised through Bayesian search targeting macro F1 score. Model interpretability was assessed using Grad CAM heatmaps reviewed by dental experts. RESULTS:EfficientNetV2 M showed the best performance for the 10-year threshold (accuracy: 0.935), DenseNet 169 for the 12-year threshold (accuracy: 0.945) and Xception for the remaining thresholds (accuracy: 14-year: 0.924; 16-year: 0.947; 18-year: 0.906; 21-year: 0.890). Errors clustered near age cut offs and increased at higher thresholds. Grad CAM highlighted posterior dento alveolar regions associated with root development. CONCLUSION:These results support convolutional neural network based orthopantomogram analysis as a judicial decision support approach and guided model selection for large scale evaluation.
Many scientific fields of study use formally established reporting standards to foster research and experimental design, transparency, replicability, peer review, and student training. Examples include CONSORT in medicine, the What Works Clearinghouse in education, JARS in psychology. Such standards yield agreement on study reporting and evaluation, even if using different methodologies. CHI has not adopted reporting standards. Like other fields, CHI has seen an increased number of low-quality submissions and reviews fueled by AI. This panel’s objective is to discuss advantages and barriers of adopting reporting standards for SIGCHI. Panelists include representatives with significant experience creating, adopting and operationalizing reporting standards in adjacent fields: software engineering, CS education, and Programming Languages. The panel will include an overview of the history of reporting standards, a live demo of a standards-based peer review system, discussions of opportunities, challenges, limitations for SIGCHI reporting standards, and an interactive discussion between attendees and panelists.
In this work, we contribute the first approach to solve infinite-horizon discounted general-utility Markov decision processes (GUMDPs) in the single-trial regime, i.e., when the agent's performance is evaluated based on a single trajectory. First, we provide some fundamental results regarding policy optimization in the single-trial regime, investigating which class of policies suffices for optimality, casting our problem as a particular MDP that is equivalent to our original problem, as well as studying the computational hardness of policy optimization in the single-trial regime. Second, we show how we can leverage online planning techniques, in particular a Monte-Carlo tree search algorithm, to solve GUMDPs in the single-trial regime. Third, we provide experimental results showcasing the superior performance of our approach in comparison to relevant baselines.