BACKGROUND:The extensive volume of biomedical scientific literature requires efficient methods for retrieving relevant documents based on semantic technologies and biomedical concepts. While embedding-based methods have shown improvements over traditional keyword-based methods, the integration of domain-specific terminologies like Medical Subject Headings (MeSH) into these models remain underexplored. METHODS:This study compares three hybrid methods that integrate MeSH-based annotations with document embeddings (called "pre-annotation", "post-annotation" and"post-reduction"). We benchmark these hybrid methods against the following traditional methods: TF-IDF, standard neural embeddings (Word2Vec, fastText, Doc2Vec) and publicly available transformer-based models (BioBERT, SciBERT, SPECTER, SapBERT), using cosine similarity and Word Mover's Distance (WMD) as evaluation metrics. The benchmark experiments are based on the RELISH corpus, a manually curated dataset of PubMed articles where experts have labeled pairs of documents with regards to their relevance to each other, providing a 2-class (relevant vs. non-relevant) as well as a 3-class (relevant, partially relevant, non-relevant) judgment. RESULTS:Transformer-based models, particularly fine-tuned BioBERT and SciBERT align best with the expert judgements after fine-tuning. Among non-transformer methods, Doc2Vec and MeSH-based hybrid methods also perform well, demonstrating the benefits from combining structured biomedical vocabularies with embedding methods. Our experiments deliver extensive results showing that the baseline performance of 76-78% precision at position 5 can be achieved through almost all approaches, improvements of 2-4% with MeSH concepts can be achieved, but performances up to 90% is left to the fine-tuned large-scale public models. CONCLUSION:The performance gains from the integration of concepts may be underwhelming, however the benefits lie in the successful integration and benchmarking of structured vocabularies with embedding methods, the applicability of these techniques to aligning literature with other data sources via a controlled vocabulary and the potential for stronger performance on tasks and corpora where concept-based resources are better suited. All experiments have been conserved as a Dockerized pipeline, making the full benchmarking workflow reproducible and supporting future research in biomedical document retrieval.
BACKGROUND:Safe and accurate sharing of healthcare information is essential for good health and well-being. Information must be exchanged among multiple parties, making the transformation between different information structures unavoidable. It has been suggested that such a transformation could be automated if information were sufficiently structured. This study aims to explore whether terminology binding to SNOMED CT can act as a bridge between different information structures. METHODS:This study employed a mixed-methods approach using two information structures from the breast cancer pathology domain. Pairs of semantically equivalent user interface terms from these structures were identified by a clinician and verified by two informaticians. The terms were considered semantically equivalent when a clinician would have used them to convey the same information. The pairs of SNOMED CT concepts bound to each of the interface terms were then compared and categorised as transformable, partially transformable, or non-transformable. Non-transformable concept pairs were analysed qualitatively, using an inductive content analysis approach, to identify underlying causes. RESULTS:73 term pairs were identified. In 70% of these, the user interface terms were terminology bound to the same SNOMED CT concept. Among term pairs with terms bound to different concepts, 9% were fully transformable, 27% partially transformable, and 64% non-transformable. All is-a-related concept pairs were either fully or partially transformable, whereas concept pairs from different hierarchies were never transformable. Major obstacles within the same hierarchy included the use of primitive concepts and nearly identical sibling concepts. For concept pairs across different hierarchies, divergent terminology-binding strategies-such as handling of integers and "other" values-hindered transformability. Additional issues were observed with local user interface terms and SNOMED CT translations. CONCLUSIONS:Although concordance between the two structures was relatively high (70%), transformability among semantically equivalent term pairs bound to different SNOMED CT concepts was low (9%). This can be improved on the terminology-binding side through guidelines and shared terminology bindings, and on the ontology side by increasing the proportion of sufficiently defined concepts and ensuring the quality of descriptions and translations. These findings are significant for practitioners seeking to enhance interoperability, as well as for SNOMED International and researchers focusing on terminology quality and transformation approaches.
Unstructured clinical narratives in electronic health records contain essential information for healthcare delivery and research. However, the presence of personally identifiable information poses significant privacy risks, which limit the secondary use of data. Therefore, reliable automated de-identification is a prerequisite for the reuse of clinical texts. This study aims to evaluate different strategies for Spanish clinical text de-identification by assessing the performance of language models on synthetic and real-world datasets. This study presents a comparative evaluation of de-identification approaches using two datasets in Spanish: MEDDOCAN, a publicly available synthetic corpus, and ObstEHR, real-world clinical narratives from an obstetrics department. Three de-identification strategies were assessed: (i) inference with a pre-trained task-specific model used as-is, (ii) prompt-based inference using locally deployed large language models (LLMs), and (iii) fine-tuned task-specific models. Model performance was evaluated using precision, recall, and F1 score on test sets. In addition, a text preservation metric was introduced to assess prompt-based LLMs, and the impact of training set size was analyzed using progressively larger subsets of annotated ObstEHR data. Models without fine-tuning and prompt-based LLMs showed limited performance, with macro-averaged F1 scores ranging from 0.14 to 0.5 on ObstEHR and from 0.19 to 0.5 on MEDDOCAN. Fine-tuned models achieved higher performance, reaching macro-averaged F1 scores of up to 0.956. Learning curve analyses showed consistently high precision and gradual improvements in recall as the amount of training data increased, with high performance achieved with moderate amounts of annotated data. Fine-tuning-based approaches outperform models without task-specific adaptation and prompt-based strategies. Despite their flexibility in generative settings, LLM-based prompt strategies show limited reliability and information preservation in clinical de-identification. Prompt-based LLMs make slight textual modifications that cause token misalignment, leading to a subsequent decrease in evaluation metrics. Therefore, domain-specific fine-tuning remains the most effective strategy for real-world clinical text de-identification.
Abstract Background Behavioral and social science research (BSSR) is essential for understanding human behavior and informing interventions, policy, and health outcomes; however, such research faces persistent challenges related to fragmented knowledge, imprecise and inconsistent terminology, and limited interoperability across studies. Ontologies provide a promising solution by enabling standardized, machine-readable representations of concepts and relationships to support data integration, knowledge synthesis, and reproducibility. Results We present an initial overview of the ACCELERATE-BASSO Research Network, an NIH-supported consortium established to advance BSSR through ontology-driven approaches. Specifically, we describe the role of the Dissemination and Coordination Center (DCC) and the Best Practices Working Group (BPWG) in supporting a network of projects focusing on accelerating BSSR through ontology development and use. Based on the consortium’s early experiences, we summarize emerging ontology-driven practices, current interoperability efforts, and common methodological considerations identified across the participating projects. Rather than proposing a formal consensus framework, this work provides an initial consortium perspective on current ontology development activities and shared lessons learned. We also discuss the potential role of ontologies in supporting AI-related applications in BSSR. Conclusions This paper provides an initial consortium perspective on ontology-driven approaches for advancing BSSR. By synthesizing early experiences, emerging practices, and shared challenges across multiple projects, it offers insights to inform future ontology development, interoperability, and community consensus while laying the foundation for more systematic evaluation and broader adoption within the BSSR community.
Knowledge Graph Embeddings are increasingly used in biomedical informatics to support similarity assessment, clustering, and knowledge discovery. Despite strong performance in link prediction, recent studies show that numerical proximity in embedding spaces does not always reflect meaningful semantic similarity. LISE (Logic-based Interactive Similarity Explainer) was introduced to expose shared semantic properties among clustered RDF resources and incorporate user feedback when evaluating cluster coherence. This work extends the previous system to LISE1 by integrating Large Language Models for natural-language explanation and investigating whether user-derived relevance signals can actively influence embedding generation, improving the semantic adequacy of similarity-based clustering. The human satisfaction with LISE1 refinement process has been evaluated in a sample of real users. We replaced LISE’s template-based verbalization component with a Gemini 2.5 Flash module capable of generating human-readable, path-level explanations of logical Common Subsumers. This resolves previous LISE limitations related to granularity and anaphora resolution, enabling reliable sentence-level user evaluations. To assess whether user preferences can guide the embedding process, we simulated feedback on 1,280 DrugBank-derived triples and evaluated two custom pyRDF2Vec sampling strategies: Predicate Relevance Weight and Predicate-Object Relevance Weight. Relevance weights were learned via a random forest regressor trained on simulated user scores. Predicate-level weighting increased the presence of user-preferred predicates in the most cohesive clusters, with all predicates showing positive or neutral deviation under learned weights. By contrast, predicate-object weighting exhibited limited sensitivity, with most pairs showing unchanged frequency regardless of weight assignment. Average deviation metrics confirm that predicate-level adjustments redirect clustering more effectively toward semantically meaningful biomedical information. User studies involving 23 participants further showed greater satisfaction with clusters refined using predicate-level weighting strategies and reveal a general improvement in users’ scores after embedding refinement. User-informed predicate weighting can successfully influence embedding-based clustering, improving alignment with semantically relevant biomedical properties. Predicate-object adjustments provide minimal benefit.
