Current image captioning systems lack the ability to link descriptive text to specific visual elements, making their outputs difficult to verify. While recent approaches offer some grounding capabilities, they cannot track object identities across multiple references or ground both actions and objects simultaneously. We propose a novel ID-based grounding system that enables consistent object reference tracking and action-object linking. We present GroundCap, a dataset containing 52,016 images from 77 movies, with 334 human-annotated and 52,016 automatically generated captions. Each caption is grounded on detected objects (132 classes) and actions (51 classes) using a tag system that maintains object identity while linking actions to the corresponding objects. Our approach features persistent object IDs for reference tracking, explicit action-object linking, and the segmentation of background elements through K-means clustering. We propose gMETEOR, a composite metric that jointly measures grounding accuracy and language quality, and establish baseline performance by fine-tuning Pixtral-12B and Qwen2.5-VL 7B on GroundCap. Human evaluation demonstrates our approach’s effectiveness in producing verifiable descriptions with coherent object references.

BACKGROUND:Placebo analgesia is a widely studied clinical phenomenon, yet placebo responses vary widely across individuals. Prior research has identified biopsychosocial factors that determine the likelihood of an individual to respond to placebo, yet generalizability and ecological validity in those studies have been limited due to the inability to account for dynamic personal and treatment effects. METHODS:We assessed fine-tuned large language models (LLMs) for the prediction of placebo responses in chronic low-back pain using contextual features extracted from patient interviews, as they speak about their lifestyle, pain, and treatment history. Interviews were conducted as part of two RCTs designed to study the placebo effect. These interviews were collected after treatment in the first trial (discovery cohort) and prior to treatment in the second trial (validation cohort). RESULTS:Semantic features extracted with LLMs can predict which individuals respond to a placebo, with an accuracy of 74% in unseen data, and validating with 70% accuracy in an independent cohort. Furthermore, in contrast to previous work, LLMs eliminated the need for pre-selecting search terms, enabling a fully data-driven approach, and provided interpretable insights into psychosocial factors underlying placebo responses. CONCLUSIONS:These findings expand on prior research by integrating state-of-art NLP techniques to address limitations in interpretability and context sensitivity of the traditional methods in related work. This method highlights the role of language models to link language and psychological states, paving the way for a deeper quantitative exploration of biopsychosocial phenomena, and to understand how they relate to treatment outcomes. SIGNIFICANCE STATEMENT:This study paves the way for a deeper yet quantitative exploration of biopsychosocial phenomena through language, and to understand how they relate to treatment outcomes, namely placebo. In this case it highlights nuanced linguistic patterns linked to responder status, which tap into semantic dimensions such as "anxiety," "resignation," and "hope".

Visual storytelling models that correctly ground entities in images may still hallucinate semantic relationships, generating incorrect dialogue attribution, character interactions, or emotional states. We introduce StoryMovie, a dataset of 1,757 stories aligned with movie scripts and subtitles through LCS matching. Our alignment pipeline synchronizes screenplay dialogue with subtitle timestamps, enabling dialogue attribution by linking character names from scripts to temporal positions from subtitles. Using this aligned content, we generate stories that maintain visual grounding tags while incorporating authentic character names, dialogue, and relationship dynamics. We fine-tune Qwen Storyteller3 on this dataset, building on prior work in visual grounding and entity re-identification. Evaluation using DeepSeek V3 as judge shows that Storyteller3 achieves an 89.9 Storyteller3 achieves 48.5

Visual storytelling systems struggle to maintain character identity across frames and link actions to appropriate subjects, frequently leading to referential hallucinations. These issues can be addressed through grounding of characters, objects, and other entities on the visual elements. We propose StoryReasoning, a dataset containing 4,178 stories derived from 52,016 movie images, with both structured scene analyses and grounded stories. Each story maintains character and object consistency across frames while explicitly modeling multi-frame relationships through structured tabular representations. Our approach features cross-frame object re-identification using visual similarity and face recognition, chain-of-thought reasoning for explicit narrative modeling, and a grounding scheme that links textual elements to visual entities across multiple frames. We establish baseline performance by fine-tuning Qwen2.5-VL 7B, creating Qwen Storyteller, which performs end-to-end object detection, re-identification, and landmark detection while maintaining consistent object references throughout the story. Evaluation demonstrates a reduction from 4.06 to 3.56 (-12.3

Clinical chronic pain assessment often relies on standardized self-report questionnaires, which constrain patients’ narratives and are frequently reported as difficult to interpret. Adapting these questionnaires across populations and cultures is costly and time-consuming, and minor ad-hoc adaptations can have major consequences in scoring validity. In this study, we investigated whether patients’ personal descriptions of their pain, i.e., their language of pain, can predict scores from standard clinical self-report measures, including assessments of pain intensity, positive and negative affect, catastrophizing, depression, and anxiety. These are crucial dimensions of the chronic pain experience. To model these predictive relations, we developed a suite of NLP pipelines, including psycholinguistic feature extraction, language model encoding, and large language model prompting, emphasizing locally hosted solutions addressing privacy concerns w.r.t. clinical data. We systematically evaluated the robustness and generalization of these pipelines across three datasets of language of pain, encompassing transcriptions of chronic pain interviews varying in topics, format, length, language, and chronic pain conditions, mirroring challenges in the clinical practice. Our results and analyses systematize the strengths and limitations of each pipeline and lay the foundation for selecting predictive models based on clinical targets and evaluation scenarios.

Placebo analgesia in chronic pain is a widely studied clinical phenomenon, where expectations about the effectiveness of a treatment can result in substantial pain relief when using an inert treatment agent. While placebos offer an opportunity for non-pharmacological treatment in chronic pain, not everyone demonstrates an analgesic response. Prior research has identified biopsychosocial factors that determine the likelihood of an individual to respond to a placebo, yet generalizability and ecological validity in those studies have been limited due to the inability to account for dynamic personal and treatment effects-which are well-known to play a role. Here, we assessed the potential of using fine-tuned large language models (LLMs) to predict placebo responders in chronic low-back pain using contextual features extracted from patient interviews, as they speak about their lifestyle, pain, and treatment history. We re-analyzed data from two clinical trials where individuals performed open-ended interviews and used these to develop a predictive model of placebo response. Our findings demonstrate that semantic features extracted with LLMs accurately predicted placebo responders, achieving a classification accuracy of 74% in unseen data, and validating with 70% accuracy in an independent cohort. Further, LLMs eliminated the need for pre-selecting search terms or to use dictionary approaches, enabling a fully data-driven approach. This LLM method further provided interpretable insights into psychosocial factors underlying placebo responses, highlighting nuanced linguistic patterns linked to responder status, which tap into semantic dimensions such as "anxiety," "resignation," and "hope." These findings expand on prior research by integrating state-of-art NLP techniques to address limitations in interpretability and context sensitivity of standard methods like bag-of-words and dictionary-based approaches. This method highlights the role of language models to link language and psychological states, paving the way for a deeper yet quantitative exploration of biopsychosocial phenomena, and to understand how they relate to treatment outcomes, including placebo.

Visual storytelling systems, particularly large vision-language models, struggle to maintain character and object identity across frames, often failing to recognize when entities in different images represent the same individuals or objects, leading to inconsistent references and referential hallucinations. This occurs because models lack explicit training on when to establish entity connections across frames. We propose a contrastive reinforcement learning approach that trains models to discriminate between coherent image sequences and stories from unrelated images. We extend the Story Reasoning dataset with synthetic negative examples to teach appropriate entity connection behavior. We employ Direct Preference Optimization with a dual-component reward function that promotes grounding and re-identification of entities in real stories while penalizing incorrect entity connections in synthetic contexts. Using this contrastive framework, we fine-tune Qwen Storyteller (based on Qwen2.5-VL 7B). Evaluation shows improvements in grounding mAP from 0.27 to 0.31 (+14.8 Pronoun grounding accuracy improved across all pronoun types except “its”, and cross-frame character and object persistence increased across all frame counts, with entities appearing in 5 or more frames advancing from 29.3 Well-structured stories, containing the chain-of-thought and grounded story, increased from 79.1

This work describes a comprehensive approach for the automatic assessment of cognitive decline from spontaneous speech in the context of the PROCESS Challenge 2025. Based on our previous experience on the use of speech and text-derived biomarkers for disease detection, we evaluate here the use of knowledge-based acoustic and text-based feature sets, as well as LLM-based macro-descriptors, and multiple neural representations (e.g., Longformer, ECAPA-TDNN, and Trillsson embeddings). The combination of these feature sets with different classifiers resulted in a large pool of systems, from which, those providing the best balance between train, development, and individual class performance were selected for model ensembling. Our final best-performing systems correspond to combinations of models that are complementary to each other, relying on acoustic and textual information from the three clinical tasks provided in the challenge dataset.

Creating engaging narratives from visual data is crucial for automated digital media consumption, assistive technologies, and interactive entertainment. This survey covers methodologies used in the generation of these narratives, focusing on their principles, strengths, and limitations. The survey also covers tasks related to automatic story generation, such as image and video captioning, and Visual Question Answering. These tasks share common challenges with Visual Story Generation (VSG) and have served as inspiration for the techniques used in the field. We analyze the main datasets and evaluation metrics, providing a critical perspective on their limitations.

Current image captioning systems lack the ability to link descriptive text to specific visual elements, making their outputs difficult to verify. While recent approaches offer some grounding capabilities, they cannot track object identities across multiple references or ground both actions and objects simultaneously. We propose a novel ID-based grounding system that enables consistent object reference tracking and action-object linking. We present GroundCap, a dataset containing 52,016 images from 77 movies, with 344 human-annotated and 52,016 automatically generated captions. Each caption is grounded on detected objects (132 classes) and actions (51 classes) using a tag system that maintains object identity while linking actions to the corresponding objects. Our approach features persistent object IDs for reference tracking, explicit action-object linking, and the segmentation of background elements through K-means clustering. We propose gMETEOR, a metric combining caption quality with grounding accuracy, and establish baseline performance by fine-tuning Pixtral-12B and Qwen2.5-VL 7B on GroundCap. Human evaluation demonstrates our approach's effectiveness in producing verifiable descriptions with coherent object references.

This work describes our group's submission to the PROCESS Challenge 2024, with the goal of assessing cognitive decline through spontaneous speech, using three guided clinical tasks. This joint effort followed a holistic approach, encompassing both knowledge-based acoustic and text-based feature sets, as well as LLM-based macrolinguistic descriptors, pause-based acoustic biomarkers, and multiple neural representations (e.g., LongFormer, ECAPA-TDNN, and Trillson embeddings). Combining these feature sets with different classifiers resulted in a large pool of models, from which we selected those that provided the best balance between train, development, and individual class performance. Our results show that our best performing systems correspond to combinations of models that are complementary to each other, relying on acoustic and textual information from all three clinical tasks.

Objectives: This study aims to systematically review the literature on the computational processing of the language of pain, whether generated by patients or physicians, identifying current trends and challenges. Methods: Following the PRISMA guidelines, a comprehensive literature search was conducted to select relevant studies on the computational processing of the language of pain and answer pre-defined research questions. Data extraction and synthesis were performed to categorize selected studies according to their primary purpose and outcome, patient and pain population, textual data, computational methodology, and outcome targets. Results: Physician-generated language of pain, specifically from clinical notes, was the most used data. Tasks included patient diagnosis and triaging, identification of pain mentions, treatment response prediction, biomedical entity extraction, correlation of linguistic features with clinical states, and lexico-semantic analysis of pain narratives. Only one study included previous linguistic knowledge on pain utterances in their experimental setup. Most studies targeted their outcomes for physicians, either directly as clinical tools or as indirect knowledge. The least targeted stage of clinical pain care was self-management, in which patients are most involved. The least studied dimensions of pain were affective and sociocultural. Only two studies measured how physician performance on clinical tasks improved with the inclusion of the proposed algorithm. Discussion: This study found that future research should focus on analyzing patient-generated language of pain, developing patient-centered resources for self-management and patient-empowerment, exploring affective and sociocultural aspects of pain, and measuring improvements in physician performance when aided by the proposed tools.

From the perspective of a dialog system, the identification of the intention behind the segments in a dialog is important, as it provides cues regarding the information present in the segments and how they should be interpreted. The ISO 24617-2 standard for dialog act annotation defines a hierarchically organized set of general-purpose communicative functions that correspond to different intentions that are relevant in the context of a dialog. In this paper, we explore the automatic recognition of these functions. To do so, we propose to adapt existing approaches to dialog act recognition, so that they can deal with the hierarchical classification problem. More specifically, we propose the use of an end-to-end hierarchical network with cascading outputs and maximum a posteriori path estimation to predict the communicative function at each level of the hierarchy, preserve the dependencies between the functions in the path, and decide at which level to stop. Additionally, we rely on transfer learning processes to address the data scarcity problem. Our experiments on the DialogBank show that this approach outperforms both flat and hierarchical approaches based on multiple classifiers and that each of its components plays an important role in the recognition of general-purpose communicative functions.

We demonstrate a proof-of-concept for the analysis of the language of chronic pain for pain intensity estimation. Importantly, we show that focus on specific words/themes is especially correlated with specific pain intensity categories. We interviewed chronic pain patients and collected demographic and clinical data. 65 patients (40 females), averaging $\mathbf{56.4} \pm \mathbf{12.7}$ years of age, participated in the study. Patients reported their current pain intensity on a Visual Analogue Scale, which we discretized into 3 classes: mild, moderate, and severe pain. We extracted language features from the transcribed interview of each patient and used them to classify their pain intensity category. We measured performance with the weighted $\mathbf{F}_{\mathbf{1}}$ score. Finally, we analyzed potential confounding variables for internal validity. The best performing model was the Support Vector Machine with an Early Fusion of select language features, with an $\mathbf{F}_{\mathbf{1}}$ of 0.60, improving 39.5% upon the baseline. Patients with mild pain focused more on verbs, whilst moderate and severe pain patients focused on adverbs, and nouns and adjectives, respectively. We show that language features from patient narratives indeed convey information relevant for pain intensity estimation, and that our models can take advantage of that.

Reported experiences of chronic pain may convey qualities relevant to the exploration of this private and subjective experience. We propose this exploration by means of the Reddit Reports of Chronic Pain (RRCP) dataset. We define and validate the RRCP for a set of subreddits related to chronic pain, identify the main concerns discussed in each subreddit, model each subreddit according to their main concerns, and compare subreddit models. The RRCP dataset comprises 86,537 submissions from 12 subreddits related to chronic pain (each related to one pathological background). Each RRCP subreddit was found to have various main concerns. Some of these concerns are shared between multiple subreddits (e.g., the subreddit Sciatica semantically entails the subreddit backpain in their various concerns, but not the other way around), whilst some concerns are exclusive to specific subreddits (e.g., Interstitialcystitis and CrohnsDisease). Our analysis details each of these concerns and their (dis)similarity relations. Although limited by the intrinsic qualities of the Reddit platform, to the best of our knowledge, this is the first research work attempting to model the linguistic expression of various chronic pain-inducing pathologies and comparing these models to identify and quantify the similarities and differences between the corresponding emergent, chronic pain experiences.

The development of artificial agents able to learn through dialog without domain restrictions has the potential to allow machines to learn how to perform tasks in a similar manner to humans and change how we relate to them. However, research in this area is practically nonexistent. In this paper, we identify the modifications required for a dialog system to be able to learn from the dialog and propose generic approaches that can be used to implement those modifications. More specifically, we discuss how knowledge can be extracted from the dialog, used to update the agent's semantic network, and grounded in action and observation. This way, we hope to raise awareness for this subject, so that it can become a focus of research in the future.

Embodied Cognition (EC) states that semantics is encoded in the brain as firing patterns of neural circuits, which are learned according to the statistical structure of human multimodal experience. However, each human brain is idiosyncratically biased, according to its subjective experience, making this biological semantic machinery noisy with respect to semantics inherent to media, such as music and language. We propose to represent media semantics using low-dimensional vector embeddings by jointly modeling the functional Magnetic Resonance Imaging (fMRI) activity of several brains via Generalized Canonical Correlation Analysis (GCCA). We evaluate the semantic richness of the resulting latent space in appropriate semantic classification tasks: music genres and language topics. We show that the resulting unsupervised representations outperform the original high-dimensional fMRI voxel spaces in these downstream tasks while being more computationally efficient. Furthermore, we show that joint modeling of several subjects increases the semantic richness of the learned latent vector spaces as the number of subjects increases. Quantitative results and corresponding statistical significance testing demonstrate the instantiation of music and language semantics in the brain, thereby providing further evidence for multimodal embodied cognition as well as a method for extraction of media semantics from multi-subject brain dynamics.

Active Learning, that is, assigning the responsibility of learning to the students, is an important tool in education as it makes the students become engaged in and think about the things they do. A similar concept was adopted in the context of Machine Learning as a means to reduce the annotation effort by selecting the examples that are most relevant or provide more information at a given time. Most studies on this subject focus on the learner’s performance. However, in interactive scenarios, the teacher’s experience is also a relevant aspect, as it affects their willingness to interact with artificial learners. In this paper, we address that aspect by performing a case study in a language grounding scenario, in which humans have to engage in dialog with a learning agent and teach it how to recognize observations of certain objects. Overall, the results of our experiments show that humans prefer to interact with an active learner, as it seems more intelligent, gives them a better perception of its knowledge, and makes the dialog more natural and enjoyable.

The computer vision community has seen a shift from convolutional-based to pure transformer architectures for both image and video tasks. Training a transformer from zero for these tasks usually requires a lot of data and computational resources. Video Swin Transformer (VST) is a pure-transformer model developed for video classification which achieves state-of-the-art results in accuracy and efficiency on several datasets. In this paper, we aim to understand if VST generalizes well enough to be used in an out-of-domain setting. We study the performance of VST on two large-scale datasets, namely FCVID and Something-Something using a transfer learning approach from Kinetics-400, which requires around 4x less memory than training from scratch. We then break down the results to understand where VST fails the most and in which scenarios the transfer-learning approach is viable. Our experiments show an 85\% top-1 accuracy on FCVID without retraining the whole model which is equal to the state-of-the-art for the dataset and a 21\% accuracy on Something-Something. The experiments also suggest that the performance of the VST decreases on average when the video duration increases which seems to be a consequence of a design choice of the model. From the results, we conclude that VST generalizes well enough to classify out-of-domain videos without retraining when the target classes are from the same type as the classes used to train the model. We observed this effect when we performed transfer-learning from Kinetics-400 to FCVID, where most datasets target mostly objects. On the other hand, if the classes are not from the same type, then the accuracy after the transfer-learning approach is expected to be poor. We observed this effect when we performed transfer-learning from Kinetics-400, where the classes represent mostly objects, to Something-Something, where the classes represent mostly actions.

The language of pain is a sub-language used to describe a subjective, private, and painful experience. In the clinical assessment and management of chronic pain, which is not as straightforward as acute pain, verbal communication is key to convey relevant information to health professionals that would otherwise not be accessible, namely, intrinsic qualities of the painful experience and that of the patient. We raise the hypothesis of applying Natural Language Processing techniques to transcribed verbal descriptions of chronic pain, to capture that information in the form of linguistic features that characterize and quantify the experience of pain of each patient. Furthermore, we demonstrate the application of these features for base-pathology prediction, specifically regarding the diagnosis of rheumatoid arthritis and spondyloarthritis. A dataset of verbal descriptions was collected for this work, considering 85 patients. The descriptions were obtained by having each patient freely answer to an interview of seven questions. The dataset was pre-processed, and features were extracted, which were then fed into binary classification machine learning models. We obtained an accuracy of 79%, in a Leave-One-Out cross-validation fashion. Based on an extensive experimental setup, we conclude that the computational analysis of the language of pain can potentially extract useful information to aid health professionals, in this case, focusing on base-pathology prediction. We also conclude on which semantic features provided more useful information for the task (distribution of pain on the body), and which did not.