Optical coherence tomography (OCT) is an advanced retinal imaging technique that enables non-invasive cross-sectional visualization of the retina, playing a crucial role in ophthalmology for detecting various macular lesions. While deep learning has shown promise in OCT image analysis, existing studies have primarily focused on broad, image-level disease diagnosis. This study introduces the Assistive Diagnosis Framework for OCT (ADF-OCT), which utilizes a dataset of over one million macular OCT images to construct a multi-label diagnostic model for common macular lesions and a medical report generation module. Our innovative Multi-frame Medical Images Distillation method effectively translates study-level multi-label annotations into image-level annotations, thereby enhancing diagnostic performance without additional annotation information. This approach significantly improves diagnostic accuracy for multi-label classification, achieving an impressive AUROC of 0.9891 with best performance macro F1 of 0.8533 and accuracy of 0.9411. By refining the feature fusion strategy in multi-frame medical imaging, our framework substantially enhances the generation of medical reports for OCT B-scans, surpassing current solutions. This research presents an advanced development pipeline that utilizes existing clinical datasets to provide more accurate and comprehensive artificial intelligence-assisted diagnoses for macular OCT.

Pretrained language models have become the dominant approach in natural language processing (NLP). However, the notable bias between the pretraining and fine-tuning stages remains unresolved when applied to downstream tasks. To address this issue, we propose RLPT, a task-adaptive reinforcement learning based post-training framework. Established on the observation that essential and task-irrelevant tokens coexist in the input of NLP tasks, we first schedule a random token pruning strategy, fine-tuning BERT to a game-like environment model. Subsequently, we utilize the Actor-Critic architecture to build the post-trained model, which can learn to discern the essential tokens for specific tasks by observing the feedback from the environment model under different token pruning actions. Experimental results demonstrate significant performance improvements on the benchmark datasets in NLP. Specifically, on the GLUE benchmark, our RLPT approach improved the BERT base model by 2.35 points, verifying its effectiveness in reducing bias and enhancing model task adaptability.

Tabular data is the most common type of data in real-life scenarios. In this study, we propose the TabKANet model for tabular data modeling, which targets the bottlenecks in learning from numerical content. We constructed a Kolmogorov-Arnold Network (KAN) based Numerical Embedding Module and unified numerical and categorical features encoding within a Transformer architecture. TabKANet has demonstrated stable and significantly superior performance compared to Neural Networks (NNs) across multiple public datasets in binary classification, multi-class classification, and regression tasks. Its performance is comparable to or surpasses that of Gradient Boosted Decision Tree models (GBDTs). Our code is publicly available on GitHub: https://github.com/AI-thpremed/TabKANet.

The accurate segmentation and quantification of retinal fluid in Optical Coherence Tomography (OCT) images are crucial for the diagnosis and treatment of ophthalmic diseases such as age-related macular degeneration. However, the accurate segmentation of retinal fluid is challenging due to significant variations in the size, position, and shape of fluid, as well as their complex, curved boundaries. To address these challenges, we propose a novel multi-scale feature fusion attention network (FNeXter), based on ConvNeXt and Transformer, for OCT fluid segmentation. In FNeXter, we introduce a novel global multi-scale hybrid encoder module that integrates ConvNeXt, Transformer, and region-aware spatial attention. This module can capture long-range dependencies and non-local similarities while also focusing on local features. Moreover, this module possesses the spatial region-aware capabilities, enabling it to adaptively focus on the lesions regions. Additionally, we propose a novel self-adaptive multi-scale feature fusion attention module to enhance the skip connections between the encoder and the decoder. The inclusion of this module elevates the model’s capacity to learn global features and multi-scale contextual information effectively. Finally, we conduct comprehensive experiments to evaluate the performance of the proposed FNeXter. Experimental results demonstrate that our proposed approach outperforms other state-of-the-art methods in the task of fluid segmentation.

The electronic medical record management system plays a crucial role in clinical practice, optimizing the recording and management of healthcare data. To enhance the functionality of the medical record management system, this paper develops a customized schema designed for ophthalmic diseases. A multi-modal knowledge graph is constructed, which is built upon expert-reviewed and de-identified real-world ophthalmology medical data. Based on this data, we propose an auxiliary diagnostic model based on a contrastive graph attention network (CGAT-ADM), which uses the patient’s diagnostic results as anchor points and achieves auxiliary medical record diagnosis services through graph clustering. By implementing contrastive methods and feature fusion of node types, text, and numerical information in medical records, the CGAT-ADM model achieved an average precision of 0.8563 for the top 20 similar case retrievals, indicating high performance in identifying analogous diagnoses. Our research findings suggest that medical record management systems underpinned by multimodal knowledge graphs significantly enhance the development of AI services. These systems offer a range of benefits, from facilitating assisted diagnosis and addressing similar patient inquiries to delving into potential case connections and disease patterns. This comprehensive approach empowers healthcare professionals to garner deeper insights and make well-informed decisions.

Pathologic myopia is one of the common eye diseases, which is becoming more severe with the increase of myopia prevailing around the world. The diagnosis of pathologic myopia and its co-existing myopic maculopathy is crucial but usually not timely enough due to the lack of experienced ophthalmologists. Therefore, we propose a new dual-branch architecture to detect pathologic myopia and classify myopic maculopathy named DBA-PMC. This architecture can be applied with any feature extraction backbone and can give pathologic myopia prediction and maculopathy classification at the same time. Because pathologic myopia and myopic maculopathy labels are corelative, two branches of the DBA-PMC can mutually promote the performance of each other through comprehension of this correlation. With extensive experiments, the DBA-PMC surpasses baseline methods with Acc 99.12% for pathologic myopia prediction and mAP 86.095 % for maculopathy classification. This work can help screen and diagnose pathologic myopia and alleviate the work of ophthalmologists.

Vision computer-aided diagnostic methods have been used in early ophthalmic disease screening and diagnosis. However, the limited output formats of these methods lead to poor human-computer interaction and low clinical applicability value. Thus, ophthalmic visual question answering is worth studying. Unfortunately, no practical solutions exist before Large Language Models(LLMs). In this paper, we investigate the ophthalmic visual diagnostic interaction problem. We construct an ophthalmology large language-and-vision assistant, OphGLM, consisting of an image encoder, a text encoder, a fusion module, and an LLM module. We establish a new Chinese ophthalmic fine-tuning dataset, FundusTuning-CN, including the fundus instruction and conversation sets. Based on FundusTuning-CN, we establish a novel LLM-tuning strategy to introduce visual model understanding and ophthalmic knowledge into LLMs at a low cost and high efficiency. Leveraging the pre-training of the image encoder, OphGLM demonstrates strong visual understanding and surpasses open-source visual language models in common fundus disease classification tasks. The FundusTuning-CN enables OphGLM to surpass open-source medical LLMs in both ophthalmic knowledge and interactive capabilities. Our proposed OphGLM has the potential to revolutionize clinical applications in ophthalmology. The dataset, code, and models will be publicly available at https://github.com/ML-AILab/OphGLM.

Large multimodal language models (LMMs) have achieved significant success in general domains. However, due to the significant differences between medical images and text and general web content, the performance of LMMs in medical scenarios is limited. In ophthalmology, clinical diagnosis relies on multiple modalities of medical images, but unfortunately, multimodal ophthalmic large language models have not been explored to date. In this paper, we study and construct an ophthalmic large multimodal model. Firstly, we use fundus images as an entry point to build a disease assessment and diagnosis pipeline to achieve common ophthalmic disease diagnosis and lesion segmentation. Then, we establish a new ophthalmic multimodal instruction-following and dialogue fine-tuning dataset based on disease-related knowledge data and publicly available real-world medical dialogue. We introduce visual ability into the large language model to complete the ophthalmic large language and vision assistant (OphGLM). Our experimental results demonstrate that the OphGLM model performs exceptionally well, and it has the potential to revolutionize clinical applications in ophthalmology. The dataset, code, and models will be made publicly available at https://github.com/ML-AILab/OphGLM.