BACKGROUND:Obtaining total knee arthroplasty patient-reported outcomes for quality assessment is costly and difficult. We asked whether a large language model (LLM) could interpret electronic health record notes to differentiate patients attaining a 1-year minimum clinically important difference (MCID) for the Knee Osteoarthritis Outcome Score-Joint Replacement (KOOS-JR) from those who did not. We also investigated whether sufficient information to infer MCID achievement exists in the chart by having a blinded orthopaedic surgeon make the same determination. METHODS:In this retrospective case-control study, we selected 40 total knee arthroplasty patients who achieved 1-year KOOS-JR MCID and 40 who did not. Orthopaedic, emergency medicine, and primary care notes from zero to six months preoperatively and nine to 15 months postoperatively were deidentified. ChatGPT 3.5 (ChatGPT) interpreted these notes to determine whether the patient improved after surgery. A blinded orthopaedic surgeon classified these patients using all chart information. The sensitivity, specificity, and accuracy of ChatGPT and the surgeon's responses were calculated. RESULTS:ChatGPT classified 78 of 80 cases with 97% sensitivity, but only 33% specificity. The surgeon's assessment had 90% sensitivity and 63% specificity. Given the equal distribution of patients meeting or not meeting MCID, Chat GPT's accuracy was 65%. The surgeon's was 76%. CONCLUSIONS:ChatGPT's assessment of KOOS-JR MCID attainment had 97% sensitivity, but only 33% specificity. False positives were commonly due to the LLM not having access to, or not properly interpreting, signs of problems in the chart. This was an initial evaluation of the current ability of a general-purpose LLM to evaluate patient outcomes based on information in chart notes. An orthopaedic surgeon's assessment of the full chart suggests an opportunity to improve on this baseline performance, possibly enabling quality monitoring and identification of best practices across a large health care system. Additional work is needed to optimize model performance and confirm the utility of this approach.

To support pharmacovigilance activities in social media, innovative methods are required to detect named entities corresponding to drugs and adverse events. However, annotated resources are missing for French-language discussion forums, and manual annotation to create training datasets is a time-consuming and complex process. We propose an approach based on generative artificial intelligence to detect relevant training examples and annotate 200 user messages from forums based on our annotation guidelines. Two prompting strategies were implemented (zero-shot and few-shot) using a decoder-only large language model (LLM). A comparison was performed between model-generated and manual annotations. Results using partial match evaluation yielded a score of 0.69-0.68 for zero-shot and few-shot prompting. While drug entity detection achieved F1-scores of 0.83, adverse event detection showed limitations (F1: 0.50-0.61), suggesting challenges in disambiguating adverse events from medical conditions in informal text. These findings suggest that while LLMs show promise for drug entity detection and initial filtering of pharmacovigilance relevant messages, the AE-Condition disambiguation task requires further refinement.

Accurate medication information is important for children, as dosing errors can have severe consequences compared to adults. We propose an automated method to extract pediatric information from Summaries of Product Characteristics (SPC). We used AirOps, a commercial and no-code visual editor to implement a processing chain with large language model GPT-4o mini on 50 SPCs. The task focused on evaluating pediatric indications and we extracted relevant sentences from all segments with a 95% recall rate and 78% precision rate. The results suggest the model can reliably classify drugs according to their pediatric indications. A no-code approach made it possible to implement the task for a healthcare professional with no training in information technology.

This paper addresses the problem of correcting medical coding errors with respect to some coding recommendations. The problem consists in clustering medical codings and determining for each cluster the set of features to correct in order to maximize the financial benefits subject to coding correction effort constraints. For this purpose, we model the coding recommendation as a disjunction of hypercubes and introduce the concept of correction sets. A mixed integer linear programming model is then proposed to assign medical codes to correction sets in order to maximize the financial benefits. The miscoding is then explained by characterizing optimal clusters with association rules and coding error distribution. A case study on patient stays associated with malnutrition-related ICD codes is presented, and the performance of the proposed methodology is assessed in regard to the current coding staff practice. A significant increase in health services reimbursement is achieved with a limited number of subjects’ features reviewed. Note to Practitioners —Medical miscoding has a significant negative impact on hospitals with a financial loss for under coding and a penalty for over coding. Whether a medical review is necessary for all descriptive features of a miscoded subject? Is it possible to reduce unnecessary medical reviews without compromising the goal of increasing hospital financial benefits? This article attempts to answer these questions with a data-driven optimization approach to determine a limited number of miscoding clusters and the set of features to review for each in order to best balance the financial benefits and the medical review workload. The application to a real-life case study leads to a significant increase in hospital fiscal revenue of nearly 6,992,489.69, while reviewing only a small number of descriptive features (5293 out of 22056 features, or 24% of features). Causes are also provided for each discovered coding error subtype to ameliorate medical coders’ coding practices. Furthermore, the proposed approach allows the decision-maker to balance the cost-benefit and the requirement of public health institutions (i.e., miscoding rate).

BackgroundTo mitigate safety concerns, regulatory agencies must make informed decisions regarding drug usage and adverse drug events (ADEs). The primary pharmacovigilance data stem from spontaneous reports by health care professionals. However, underreporting poses a notable challenge within the current system. Explorations into alternative sources, including electronic patient records and social media, have been undertaken. Nevertheless, social media’s potential remains largely untapped in real-world scenarios. ObjectiveThe challenge faced by regulatory agencies in using social media is primarily attributed to the absence of suitable tools to support decision makers. An effective tool should enable access to information via a graphical user interface, presenting data in a user-friendly manner rather than in their raw form. This interface should offer various visualization options, empowering users to choose representations that best convey the data and facilitate informed decision-making. Thus, this study aims to assess the potential of integrating social media into pharmacovigilance and enhancing decision-making with this novel data source. To achieve this, our objective was to develop and assess a pipeline that processes data from the extraction of web forum posts to the generation of indicators and alerts within a visual and interactive environment. The goal was to create a user-friendly tool that enables regulatory authorities to make better-informed decisions effectively. MethodsTo enhance pharmacovigilance efforts, we have devised a pipeline comprising 4 distinct modules, each independently editable, aimed at efficiently analyzing health-related French web forums. These modules were (1) web forums’ posts extraction, (2) web forums’ posts annotation, (3) statistics and signal detection algorithm, and (4) a graphical user interface (GUI). We showcase the efficacy of the GUI through an illustrative case study involving the introduction of the new formula of Levothyrox in France. This event led to a surge in reports to the French regulatory authority. ResultsBetween January 1, 2017, and February 28, 2021, a total of 2,081,296 posts were extracted from 23 French web forums. These posts contained 437,192 normalized drug-ADE couples, annotated with the Anatomical Therapeutic Chemical (ATC) Classification and Medical Dictionary for Regulatory Activities (MedDRA). The analysis of the Levothyrox new formula revealed a notable pattern. In August 2017, there was a sharp increase in posts related to this medication on social media platforms, which coincided with a substantial uptick in reports submitted by patients to the national regulatory authority during the same period. ConclusionsWe demonstrated that conducting quantitative analysis using the GUI is straightforward and requires no coding. The results aligned with prior research and also offered potential insights into drug-related matters. Our hypothesis received partial confirmation because the final users were not involved in the evaluation process. Further studies, concentrating on ergonomics and the impact on professionals within regulatory agencies, are imperative for future research endeavors. We emphasized the versatility of our approach and the seamless interoperability between different modules over the performance of individual modules. Specifically, the annotation module was integrated early in the development process and could undergo substantial enhancement by leveraging contemporary techniques rooted in the Transformers architecture. Our pipeline holds potential applications in health surveillance by regulatory agencies or pharmaceutical companies, aiding in the identification of safety concerns. Moreover, it could be used by research teams for retrospective analysis of events.

PurposeThis study aims to evaluate the performance of LLMs with various prompt engineering strategies in the context of health fact-checking.Design/methodology/approachInspired by Dual Process Theory, we introduce two kinds of prompts: Conclusion-first (System 1) and Explanation-first (System 2), and their respective retrieval-augmented variations. We evaluate the performance of these prompts across accuracy, argument elements, common errors and cost-effectiveness. Our study, conducted on two public health fact-checking datasets, categorized 10,212 claims as knowledge, anecdotes and news. To further analyze the reasoning process of LLM, we delve into the argument elements of health fact-checking generated by different prompts, revealing their tendencies in using evidence and contextual qualifiers. We conducted content analysis to identify and compare the common errors across various prompts.FindingsResults indicate that the Conclusion-first prompt performs well in knowledge (89.70%,66.09%), anecdote (79.49%,79.99%) and news (85.61%,85.95%) claims even without retrieval augmentation, proving to be cost-effective. In contrast, the Explanation-first prompt often classifies claims as unknown. However, it significantly boosts accuracy for news claims (87.53%,88.60%) and anecdote claims (87.28%,90.62%) with retrieval augmentation. The Explanation-first prompt is more focused on context specificity and user intent understanding during health fact-checking, showing high potential with retrieval augmentation. Additionally, retrieval-augmented LLMs concentrate more on evidence and context, highlighting the importance of the relevance and safety of retrieved content.Originality/valueThis study offers insights into how a balanced integration could enhance the overall performance of LLMs in critical applications, paving the way for future research on optimizing LLMs for complex cognitive tasks.Peer reviewThe peer review history for this article is available at: https://publons.com/publon/10.1108/OIR-02-2024-0111

The ability to fine-tune pre-trained deep learning models to learn how to process a downstream task using a large training set allow to significantly improve performances of named entity recognition. Large language models are recent models based on the Transformers architecture that may be conditioned on a new task with in-context learning, by providing a series of instructions or prompt. These models only require few examples and such approach is defined as few shot learning. Our objective was to compare performances of named entity recognition of adverse drug events between state of the art deep learning models fine-tuned on Pubmed abstracts and a large language model using few-shot learning. Hussain et al's state of the art model (PMID: 34422092) significantly outperformed the ChatGPT-3.5 model (F1-Score: 97.6% vs 86.0%). Few-shot learning is a convenient way to perform named entity recognition when training examples are rare, but performances are still inferior to those of a deep learning model fine-tuned with several training examples. Perspectives are to evaluate few-shot prompting with GPT-4 and perform fine-tuning on GPT-3.5.

The increasing use of ChatGPT by the general public has prompted us to assess ChatGPT's performance in health fact-checking and uncover potential biases and risks arising from its utilization. In this study, we employed two publicly accessible datasets to evaluate ChatGPT's performance. We utilized BERTopic for clustering health claims into topics and subsequently employed the gpt-3.5-turbo API for fact-checking these claims. ChatGPT's performance was appraised on multi-class (False, Mixture, Mostly-False, Mostly-True, True) and binary (True, False) levels, with a thorough analysis of its performance across various topics. ChatGPT achieved a F1-score of 0.54 and 0.64 in the multi-class task and 0.88 and 0.85 in the binary task on the two datasets, respectively. In most health topics (e.g., vaccines, Covid-19), ChatGPT's F1-score exceeded 0.8, except for specific topics, such as novel or contentious cancer treatments, which yielded a F1-score below 0.6. We scrutinized the erroneous fact-checking labels and explanations provided by ChatGPT, revealing that it may produce inaccurate results for claims with misleading intent, inaccurate information, emerging research findings, or contentious health knowledge.

The proliferation of health misinformation in recent years has prompted the development of various methods for detecting and combatting this issue. This review aims to provide an overview of the implementation strategies and characteristics of publicly available datasets that can be used for health misinformation detection. Since 2020, a large number of such datasets have emerged, half of which are focused on COVID-19. Most of the datasets are based on fact-checkable websites, while only a few are annotated by experts. Furthermore, some datasets provide additional information such as social engagement and explanations, which can be utilized to study the spread of misinformation. Overall, these datasets offer a valuable resource for researchers working to combat the spread and consequences of health misinformation.

BACKGROUND:Within a few months, the COVID-19 pandemic had spread to many countries and had been a real challenge for health systems all around the world. This unprecedented crisis has led to a surge of online discussions about potential cures for the disease. Among them, vaccines have been at the heart of the debates and have faced lack of confidence before marketing in France.OBJECTIVE:This study aims to identify and investigate the opinions of French Twitter users on the announced vaccines against COVID-19 through sentiment analysis.METHODS:This study was conducted in 2 phases. First, we filtered a collection of tweets related to COVID-19 available on Twitter from February 2020 to August 2020 with a set of keywords associated with vaccine mistrust using word embeddings. Second, we performed sentiment analysis using deep learning to identify the characteristics of vaccine mistrust. The model was trained on a hand-labeled subset of 4548 tweets.RESULTS:A set of 69 relevant keywords were identified as the semantic concept of the word "vaccin" (vaccine in French) and focused mainly on conspiracies, pharmaceutical companies, and alternative treatments. Those keywords enabled us to extract nearly 350,000 tweets in French. The sentiment analysis model achieved 0.75 accuracy. The model then predicted 16% of positive tweets, 41% of negative tweets, and 43% of neutral tweets. This allowed us to explore the semantic concepts of positive and negative tweets and to plot the trends of each sentiment. The main negative rhetoric identified from users' tweets was that vaccines are perceived as having a political purpose and that COVID-19 is a commercial argument for the pharmaceutical companies.CONCLUSIONS:Twitter might be a useful tool to investigate the arguments for vaccine mistrust because it unveils political criticism contrasting with the usual concerns on adverse drug reactions. As the opposition rhetoric is more consistent and more widely spread than the positive rhetoric, we believe that this research provides effective tools to help health authorities better characterize the risk of vaccine mistrust.

Our objective was to improve the accuracy of bacteria and resistance coding in a hospital case mix database. Data sources consisted of 50,074 files on bacteriological susceptibility tests transmitted with the HPRIM protocol from laboratory management system to electronic health record of the University hospital of Saint Etienne in July 2017. An algorithm was implemented to detect susceptibility tests containing information corresponding to codes whose addition in the case mix database was susceptible to increase the severity level of a diagnosis related group. Among 132 hospital stays fulfilling the conditions, 27 were lacking bacteria and/or resistance codes, and the tariff was increased for 9 stays, with earnings of €54,612. Analyzing Antimicrobial susceptibility tests helps to improve clinical coding and optimize the financial gain.

Since December 2019 and the first reported cases of COVID-19 in Wuhan, China, there have been 199,466,211 confirmed cases of COVID-19 in the World. The WHO defined vaccination hesitancy as one of the top ten threats to global health in 2019. Our objective was thus to identify topics and trends about COVID-19 vaccines from French web forums to understand the perception of the French population on these vaccines before the vaccination campaign started. We performed a topic model analysis on 485 web forums’ posts. 10 topics were found. We reviewed 120 posts from 6 of these 10 topics. One topic was about vaccine hesitancy, refusal, and mistrust, and two topics were related to what the users think about the government, the political and economic choices made towards this epidemic.

In previous work, we implemented a deep learning model with CamemBERT and PyTorch, and built a microservices architecture using the TorchServe serving library. Without TorchServe, inference time was three times faster when the model was loaded once in memory compared when the model was loaded each time. The preloaded model without TorchServe presented comparable inference time with the TorchServe instance. However, using a PyTorch preloaded model in a web application without TorchServe would necessitate to implement functionalities already present in TorchServe.

Artificial Intelligence (AI) has made major progress in recent years in many fields. With regard of medicine however, the utilization of AI raises numerous ethical questions, especially since newer and much more accurate algorithms function as black boxes. A trade-off must then be made between having algorithms being very accurate and effective, and algorithms that are explainable but less proficient. In this paper we address the ethical implications of utilizing black box algorithms in medicine.

A strong trend in the software industry is to merge the activities of deployment and operationalization through the DevOps approach, which in the case of artificial intelligence is called Machine Learning Operations (MLOps). We present here a microservices architecture containing the whole pipeline (frontend, backend, data predictions) hosted in Docker containers which exposes a model implemented for opinion prediction in Twitter on the COVID vaccines. This is the first description in the literature of implementing a microservice architecture using TorchServe, a library for serving Pytorch models.