Current omics repositories aim to facilitate the findability of different types of omics data in order to improve data reuse. Data reuse requires sufficient metadata about how the data was produced and processed. This is part of the “rich metadata” for which the FAIR principles advocate. Adoption of these FAIR principles will make the ever-increasing amount of omics data more Findable, Accessible, Interoperable and Reusable for machines and humans. This is especially important in the rare disease domain, which often relies on the reuse of data for analysis, given the small number of patients with a specific rare disease. We investigated the findability and reusability of omics datasets in two ways; with a generic, automated tool that evaluates the metadata of a resource for compliance with the FAIR principles and with a use-case driven assessment specific to the rare-disease domain. We chose 7 repositories to test for human and machine findability and reusability. The metadata of omics datasets presented in the various webpages of repositories on average passed only 2 of the 10 tests done by the generic FAIR Evaluator. The main reason for the failed tests was the lack of embedded metadata on the webpages of the datasets. The second method, where we searched for rare disease datasets, showed that the results were mostly false positives over all repositories. This reduces the precision of the search, which on average was 0.09 with a maximum of 0.3 and a minimum of 0 over the 7 repositories tested. FAIRifying metadata on omics repositories will allow users to find and reuse the datasets in the repositories efficiently. To achieve this, based on our analyses, we recommend 5 actions repositories could take. Steps like providing machine-readable metadata using semantic web technologies like RDF, using a fully semantic model and proper use of controlled terminologies would vastly improve the Findability and Reusability of datasets.
Abstract Background Phenotyping, the comprehensive assessment of observable characteristics, is essential for advancing medical understanding and personalised healthcare. However, traditional phenotyping methods are often manual, time-intensive, and limited in scope. To address these challenges, this work introduces the TOP Framework, a software suite that leverages a structured approach. It provides tools for the formal definition of phenotypes, their organisation into ontological classes, the creation of phenotype models for disease-specific knowledge representation, and the generation of phenotype queries for automated data retrieval and analysis. A dynamic phenotype algorithm integrates these modules to efficiently identify individuals meeting complex phenotypic criteria. The Model for End-Stage Liver Disease (MELD) score serves as a running example to illustrate the framework’s capabilities. Furthermore, this paper presents a preliminary evaluation of the TOP Framework’s user experience by means of the User Experience Questionnaire (UEQ), assessing its usability and suitability for researchers and clinicians. Results The TOP Framework includes a robust implementation of a reasoner model for deriving complex phenotypes and an automated testing module to ensure reliability. The user experience evaluation yielded generally positive results on a scale from −3 to 3 ( n = 11), with mean scores and 95% confidence intervals as follows: attractiveness, 1.50 (CI = (1.07, 1.93)); pragmatic quality, 1.38 (CI = (1.00, 1.77)); and hedonic quality, 1.40 (CI = (0.76, 2.03)). Conclusions The TOP Framework offers a novel and automated approach to phenotyping, with the potential to enhance the efficiency, scalability, and reproducibility of phenotyping studies. This advancement contributes to a deeper understanding of disease and the progression of precision medicine.
BACKGROUND:The extensive volume of biomedical scientific literature requires efficient methods for retrieving relevant documents based on semantic technologies and biomedical concepts. While embedding-based methods have shown improvements over traditional keyword-based methods, the integration of domain-specific terminologies like Medical Subject Headings (MeSH) into these models remain underexplored. METHODS:This study compares three hybrid methods that integrate MeSH-based annotations with document embeddings (called "pre-annotation", "post-annotation" and"post-reduction"). We benchmark these hybrid methods against the following traditional methods: TF-IDF, standard neural embeddings (Word2Vec, fastText, Doc2Vec) and publicly available transformer-based models (BioBERT, SciBERT, SPECTER, SapBERT), using cosine similarity and Word Mover's Distance (WMD) as evaluation metrics. The benchmark experiments are based on the RELISH corpus, a manually curated dataset of PubMed articles where experts have labeled pairs of documents with regards to their relevance to each other, providing a 2-class (relevant vs. non-relevant) as well as a 3-class (relevant, partially relevant, non-relevant) judgment. RESULTS:Transformer-based models, particularly fine-tuned BioBERT and SciBERT align best with the expert judgements after fine-tuning. Among non-transformer methods, Doc2Vec and MeSH-based hybrid methods also perform well, demonstrating the benefits from combining structured biomedical vocabularies with embedding methods. Our experiments deliver extensive results showing that the baseline performance of 76-78% precision at position 5 can be achieved through almost all approaches, improvements of 2-4% with MeSH concepts can be achieved, but performances up to 90% is left to the fine-tuned large-scale public models. CONCLUSION:The performance gains from the integration of concepts may be underwhelming, however the benefits lie in the successful integration and benchmarking of structured vocabularies with embedding methods, the applicability of these techniques to aligning literature with other data sources via a controlled vocabulary and the potential for stronger performance on tasks and corpora where concept-based resources are better suited. All experiments have been conserved as a Dockerized pipeline, making the full benchmarking workflow reproducible and supporting future research in biomedical document retrieval.
BACKGROUND:Safe and accurate sharing of healthcare information is essential for good health and well-being. Information must be exchanged among multiple parties, making the transformation between different information structures unavoidable. It has been suggested that such a transformation could be automated if information were sufficiently structured. This study aims to explore whether terminology binding to SNOMED CT can act as a bridge between different information structures. METHODS:This study employed a mixed-methods approach using two information structures from the breast cancer pathology domain. Pairs of semantically equivalent user interface terms from these structures were identified by a clinician and verified by two informaticians. The terms were considered semantically equivalent when a clinician would have used them to convey the same information. The pairs of SNOMED CT concepts bound to each of the interface terms were then compared and categorised as transformable, partially transformable, or non-transformable. Non-transformable concept pairs were analysed qualitatively, using an inductive content analysis approach, to identify underlying causes. RESULTS:73 term pairs were identified. In 70% of these, the user interface terms were terminology bound to the same SNOMED CT concept. Among term pairs with terms bound to different concepts, 9% were fully transformable, 27% partially transformable, and 64% non-transformable. All is-a-related concept pairs were either fully or partially transformable, whereas concept pairs from different hierarchies were never transformable. Major obstacles within the same hierarchy included the use of primitive concepts and nearly identical sibling concepts. For concept pairs across different hierarchies, divergent terminology-binding strategies-such as handling of integers and "other" values-hindered transformability. Additional issues were observed with local user interface terms and SNOMED CT translations. CONCLUSIONS:Although concordance between the two structures was relatively high (70%), transformability among semantically equivalent term pairs bound to different SNOMED CT concepts was low (9%). This can be improved on the terminology-binding side through guidelines and shared terminology bindings, and on the ontology side by increasing the proportion of sufficiently defined concepts and ensuring the quality of descriptions and translations. These findings are significant for practitioners seeking to enhance interoperability, as well as for SNOMED International and researchers focusing on terminology quality and transformation approaches.
Unstructured clinical narratives in electronic health records contain essential information for healthcare delivery and research. However, the presence of personally identifiable information poses significant privacy risks, which limit the secondary use of data. Therefore, reliable automated de-identification is a prerequisite for the reuse of clinical texts. This study aims to evaluate different strategies for Spanish clinical text de-identification by assessing the performance of language models on synthetic and real-world datasets. This study presents a comparative evaluation of de-identification approaches using two datasets in Spanish: MEDDOCAN, a publicly available synthetic corpus, and ObstEHR, real-world clinical narratives from an obstetrics department. Three de-identification strategies were assessed: (i) inference with a pre-trained task-specific model used as-is, (ii) prompt-based inference using locally deployed large language models (LLMs), and (iii) fine-tuned task-specific models. Model performance was evaluated using precision, recall, and F1 score on test sets. In addition, a text preservation metric was introduced to assess prompt-based LLMs, and the impact of training set size was analyzed using progressively larger subsets of annotated ObstEHR data. Models without fine-tuning and prompt-based LLMs showed limited performance, with macro-averaged F1 scores ranging from 0.14 to 0.5 on ObstEHR and from 0.19 to 0.5 on MEDDOCAN. Fine-tuned models achieved higher performance, reaching macro-averaged F1 scores of up to 0.956. Learning curve analyses showed consistently high precision and gradual improvements in recall as the amount of training data increased, with high performance achieved with moderate amounts of annotated data. Fine-tuning-based approaches outperform models without task-specific adaptation and prompt-based strategies. Despite their flexibility in generative settings, LLM-based prompt strategies show limited reliability and information preservation in clinical de-identification. Prompt-based LLMs make slight textual modifications that cause token misalignment, leading to a subsequent decrease in evaluation metrics. Therefore, domain-specific fine-tuning remains the most effective strategy for real-world clinical text de-identification.
