Nuclei segmentation and classification is a fundamental task in the field of computational pathology. Existing approaches have demonstrated outstanding performance. However, they still suffer from the efficiency problem due to the extremely huge resolution of whole slide images and the complexity of the model, which greatly hinder their viability in real-world clinical scenarios. In this study, we propose a lightweight and fast nuclei segmentation and classification model, called FKDNuSeg. FKDNuSeg employs a knowledge distillation strategy with a multi-task learning architecture using ENet as the backbone. A flawless knowledge distillation method is designed to encourage the student model to learn the correct knowledge from the teacher model while preventing it being over influenced by flawed information, especially for the minority classes. An edge detection task enhanced with a curvature module is introduced to extract more precise nuclei edges and separate the attached or overlapping nuclei. FKDNuSeg is over 10× smaller than the state-of-the-art models with only 4.11M parameters. The proposed model achieves an inference time (patch-level) of 1.38 ms on an RTX 3090 GPU. Extensive experiments on two publicly available datasets CoNSeP and PanNuke demonstrate that FKDNuSeg significantly reduces the model complexity and improves the model efficiency while maintaining considerable performance.

Personalized Federated Learning (PFL) presents a promising paradigm for mitigating privacy concerns and addressing data heterogeneity in medical images. However, the presence of data heterogeneity can introduce bias into the global model. As a result, the biased global model may degrade local performance in the personalization phase. To tackle this challenge, we propose FedLP, a novel framework that enhances global aggregation fairness and improves personalized performance. First, we propose a layer-wise aggregation strategy on the server, where it dynamically estimates each client's contribution based on the gradient discrepancies. Consider the aggregated global model may not consistently align with local distributions, we then introduce an adaptive parameter personalization process on the client side. By selectively instituting local parameters with the global model parameter with maximizing consensus optimization, clients can better retain global knowledge they need and maintain important model parameters to local data distributions. Experimental results demonstrate that FedLP outperforms state-of-the-art methods and achieves fairer performance for several medical image classification tasks.

Visual Prompt Tuning (VPT) has significantly enhanced training efficacy across various computer vision tasks. It provides a promising approach for Federated Learning (FL) to efficiently address heterogeneity by adapting pre-trained vision transformer models. However, current federated VPT methods treat prompts as single-functional and domain-specific features, neglecting their potential to capture signals from different aspects and various domains. To address this limitation, we propose a novel algorithm, pFedDP, that decouples visual prompts into Specific Prompts (SPs) and Common Prompts (CPs). By assigning distinct learning heads to SPs and CPs and leveraging orthogonal loss between the specific head and common head, our method enables the learning of both domain-specific and shared knowledge. To better promote common knowledge fusion, we further introduce an adaptive aggregation method that cleverly utilizes features gained from each client’s SPs to guide the aggregation of common heads. This can more effectively separate specific knowledge from the shared knowledge. Through extensive experiments on feature heterogeneity settings, we demonstrate that pFedDP outperforms state-of-the-art personalized FL methods, validating its superior capabilities in handling federated learning challenges. Code will be released after acceptance.

Mitosis detection is one of the fundamental tasks in computational pathology, which is extremely challenging due to the heterogeneity of mitotic cell. Most of the current studies solve the heterogeneity in the technical aspect by increasing the model complexity. However, lacking consideration of the biological knowledge and the complex model design may lead to the overfitting problem while limited the generalizability of the detection model. In this paper, we systematically study the morphological appearances in different mitotic phases as well as the ambiguous non-mitotic cells and identify that balancing the data and feature diversity can achieve better generalizability. Based on this observation, we propose a novel generalizable framework (MitDet) for mitosis detection. The data diversity is considered by the proposed diversity-guided sample balancing (DGSB). And the feature diversity is preserved by inter- and intra- class feature diversity-preserved module (InCDP). Stain enhancement (SE) module is introduced to enhance the domain-relevant diversity of both data and features simultaneously. Extensive experiments have demonstrated that our proposed model outperforms all the state-of-the-art (SOTA) approaches in several popular mitosis detection datasets in both internal and unseen test sets using point annotations only. Comprehensive ablation studies have also proven the effectiveness of the rethinking of data and feature diversity balancing. By analyzing the results quantitatively and qualitatively, we believe that our proposed model not only achieves SOTA performance but also might inspire the future studies in new perspectives. Code is available at https://github.com/linjiatai/MitDet.

Target detection is critical for unmanned aerial vehicle (UAV) applications, yet the limited computing and storage resources of UAV platforms pose significant challenges for achieving accurate and efficient detection. This study proposes a lightweight small-object detection algorithm tailored for UAV imagery, integrating lightweight modules to reduce model parameters while maintaining accuracy through streamlined computations and a simplified network architecture. The feature fusion layer is optimized by pruning the branch for large-object detection, thereby eliminating redundant computations, and incorporating an additional feature extraction module to enhance shallow feature representation. Furthermore, a logit-based knowledge distillation strategy is employed, enabling the student model to leverage high-level representations from the teacher model, achieving superior detection performance and improved generalization with fewer parameters and lower computational cost. Compared to the baseline, the proposed method significantly enhances small-object detection accuracy while reducing model complexity.

Atomic force microscopy (AFM) is essential for studying the surface properties of samples at the micro- and nanoscales. Traditional AFM scanning methods are time-consuming, particularly for obtaining high-resolution images. Compressive sensing (CS) has been utilized for fast AFM imaging. However, as the size and resolution requirements of the images increase, the measurement matrix for compressive sensing also becomes larger. Block compressive sensing (BCS) divides the image into blocks and reconstructs them with a small measurement matrix, but it is difficult to balance the imaging quality between regions. Therefore, we propose an innovative adaptive CS-AFM imaging scheme. A low-resolution image is obtained through fast scanning, and a high-resolution image is generated using bicubic interpolation. The Otsu and eight-connectivity methods detect the location of the target blocks, while the GRNN model adapts the sampling rate for it. A supplementary scan is performed on the target block, followed by reconstruction using the TVAL3 algorithm. Finally, the target region is replaced with the reconstructed high-quality target blocks. Compared to other schemes, the results demonstrate that our method excels in achieving fast, high-quality, and high-resolution imaging.

Image classification serves as a essential task of medical image processing, relying heavily on labeled data. Due to the necessity of professional medical knowledge, it is difficult to rapidly accumulate a large number of annotations through crowd-sourcing. Research on active learning has been developed to select the most valuable data within limited labeling budgets. Traditional active learning methods select samples by uncertainty measure or representative information. They may overlook the distribution characteristics and boundary information of samples in the feature space. We introduce a novel active learning approach called Active Learning by Feature Perturbation (ALFP). It evaluates samples based on the impact of feature perturbation on model decisions and the degree of perturbations. ALFP first assigns mandatory labels of each class to guide perturbations and obtain new predictions of samples. If the model prediction of sample changes, we regard it as an uncertain one. Then the perturbation distances obtained by feature perturbators are regarded as diversity vectors, which can be used to select valuable samples by clustering. Finally, we select the data points that are closest to the cluster centers in the cluster space with the same number of centers. In this paper, we conduct experiments using traditional active learning settings on classification tasks of three public medical image datasets. We compare our method with several representative active learning approaches and our ALFP obtains the state-of-the-art (SOTA) performance. It demonstrates that perturbation-based information can effectively evaluate sample value and enhance sampling strategies for active learning.

Ultrasonography plays an essential role in breast cancer diagnosis. Current deep learning based studies train the models on either images or videos in a centralized learning manner, lacking consideration of joint benefits between two different modality models or the privacy issue of data centralization. In this study, we propose the first decentralized learning solution for joint learning with breast ultrasound video and image, called FedBCD. To enable the model to learn from images and videos simultaneously and seamlessly in client-level local training, we propose a Joint Ultrasound Video and Image Learning (JUVIL) model to bridge the dimension gap between video and image data by incorporating temporal and spatial adapters. The parameter-efficient design of JUVIL with trainable adapters and frozen backbone further reduces the computational cost and communication burden of federated learning, finally improving the overall efficiency. Moreover, considering conventional model-wise aggregation may lead to unstable federated training due to different modalities, data capacities in different clients, and different functionalities across layers. We further propose a Fisher information matrix (FIM) guided Layer-wise Aggregation method named FILA. By measuring layer-wise sensitivity with FIM, FILA assigns higher contributions to the clients with lower sensitivity, improving personalized performance during federated training. Extensive experiments on three image clients and one video client demonstrate the benefits of joint learning architecture, especially for the ones with small-scale data. FedBCD significantly outperforms nine federated learning methods on both video-based and image-based diagnoses, demonstrating the superiority and potential for clinical practice. Code is released at https://github.com/tianpeng-deng/FedBCD.

Semantic segmentation for histopathology images is one of the fundamental tasks in computational pathology. Due to the high cost of pixel-level annotation acquisition, the weakly supervised semantic segmentation (WSSS) attempts to achieve information-intensive segmentation task for histopathology images to reduce the labeling effort of pathologists by leveraging image-level labels. However, traditional WSSS requires a large-scale training set with image-level labels, which still imposes considerable labeling costs on pathologists. To this end, this work proposes a Small-Scale Weakly Supervised Semantic Segmentation (SS-WSSS) approach to achieve the comparable performance only with small-scale weakly-labeled data to further reduce pathologist's labeling effort. Since histopathology images can easily generate massive unlabeled data, our SS-WSSS aims to learn with the unlabeled data to bridge the information gap. First, we propose a Single-to-Multi Prototype Similarity (S2MPS) method to generate reliable pseudo-labels for unlabeled data by measuring the similarity between single-label prototypes and multi-label feature maps. Then, we introduce a Cross-Task CoTraining (CT2) method for pseudo-supervision of models with pseudo-labels self-refinement to avoid overfitting to noisy labels. We conduct the experiment on two public datasets to demonstrate the effectiveness of our SS-WSSS. In the experiment, our method achieves comparable performance with SOTA methods only using 30% labeled data.

The fully convolutional network (FCN) has dominated salient object detection for a long period. However, the locality of CNN requires the model deep enough to have a global receptive field and such a deep model always leads to the loss of local details. In this paper, we introduce a new attention-based encoder, vision transformer, into salient object detection to ensure the globalization of the representations from shallow to deep layers. With the global view in very shallow layers, the transformer encoder preserves more local representations to recover the spatial details in final saliency maps. Besides, as each layer can capture a global view of its previous layer, adjacent layers can implicitly maximize the representation differences and minimize the redundant features, making every output feature of transformer layers contribute uniquely to the final prediction. To decode features from the transformer, we propose a simple yet effective deeply-transformed decoder. The decoder densely decodes and upsamples the transformer features, generating the final saliency map with less noise injection. Experimental results demonstrate that our method significantly outperforms other FCN-based and transformer-based methods in five benchmarks by a large margin, with an average of 12.17% improvement in terms of Mean Absolute Error (MAE).

Medical image classification is the foundational task in medical image analysis which depends on the high-quality annotated training data. Due to the high cost of medical image annotation and the strong heterogeneity of medical data, how to acquire high-quality annotation under the limited labeling budget has become a core requirement. Active learning (AL) provides a human-in-the-loop labeling scheme to reduce the labeling cost by selecting more valuable samples from unlabeled sets to be labeled. Uncertainty-based and diversity-based methods are the most mainstream active learning strategy. Recently, gradient information has been innovatively used to estimate model change as uncertainty and diversity for data sampling. However, gradient-based approaches may cause the problem of gradient vanishing or exploding by insufficient training samples. To avoid this problem, our idea is to measure the model change of each query sample in a gradient-free way. This paper proposes a deep-broad active learning (DBrAL) strategy to achieve this by utilizing the incremental learning of a broad learning system (BLS) to calculate the weight changes. DBrAL first extracts the features from the deep learner and uses them to fix the BLS. Then, incremental learning is used to quantifyă the weight change for each query sample. Since DBrAL directly establishes the relationship between query samples and the classification hyperplane, it avoids the problem of gradients, which can be more reliable. In the experiment, we demonstrate the effectiveness of our DBrAL on two public medical image classification datasets compared with several SOTA methods.

Histopathological tissue classification is a fundamental task in computational pathology. Deep learning (DL)-based models have achieved superior performance but centralized training suffers from the privacy leakage problem. Federated learning (FL) can safeguard privacy by keeping training samples locally, while existing FL-based frameworks require a large number of well-annotated training samples and numerous rounds of communication which hinder their viability in real-world clinical scenarios. In this article, we propose a lightweight and universal FL framework, named federated deep-broad learning (FedDBL), to achieve superior classification performance with limited training samples and only one-round communication. By simply integrating a pretrained DL feature extractor, a fast and lightweight broad learning inference system with a classical federated aggregation approach, FedDBL can dramatically reduce data dependency and improve communication efficiency. Five-fold cross-validation demonstrates that FedDBL greatly outperforms the competitors with only one-round communication and limited training samples, while it even achieves comparable performance with the ones under multiple-round communications. Furthermore, due to the lightweight design and one-round communication, FedDBL reduces the communication burden from 4.6 GB to only 138.4 KB per client using the ResNet-50 backbone at 50-round training. Extensive experiments also show the scalability of FedDBL on model generalization to the unseen dataset, various client numbers, model personalization and other image modalities. Since no data or deep model sharing across different clients, the privacy issue is well-solved and the model security is guaranteed with no model inversion attack risk. Code is available at https://github.com/tianpeng-deng/FedDBL.

Brain tumor segmentation (BTS) in magnetic resonance image (MRI) is crucial for brain tumor diagnosis, cancer management and research purposes. With the great success of the ten-year BraTS challenges as well as the advances of CNN and Transformer algorithms, a lot of outstanding BTS models have been proposed to tackle the difficulties of BTS in different technical aspects. However, existing studies hardly consider how to fuse the multi-modality images in a reasonable manner. In this paper, we leverage the clinical knowledge of how radiologists diagnose brain tumors from multiple MRI modalities and propose a clinical knowledge-driven brain tumor segmentation model, called CKD-TransBTS. Instead of directly concatenating all the modalities, we re-organize the input modalities by separating them into two groups according to the imaging principle of MRI. A dual-branch hybrid encoder with the proposed modality-correlated cross-attention block (MCCA) is designed to extract the multi-modality image features. The proposed model inherits the strengths from both Transformer and CNN with the local feature representation ability for precise lesion boundaries and long-range feature extraction for 3D volumetric images. To bridge the gap between Transformer and CNN features, we propose a Trans&CNN Feature Calibration block (TCFC) in the decoder. We compare the proposed model with six CNN-based models and six transformer-based models on the BraTS 2021 challenge dataset. Extensive experiments demonstrate that the proposed model achieves state-of-the-art brain tumor segmentation performance compared with all the competitors.

Skeleton‐based gait recognition models suffer from the robustness problem, as the rank‐1 accuracy varies from 90% in normal walking cases to 70% in walking with coats cases. In this work, we propose a state‐of‐the‐art robust skeleton‐based gait recognition model called Gait‐TR, which is based on the combination of spatial transformer frameworks and temporal convolutional networks. Gait‐TR achieves substantial improvements over other skeleton‐based gait models with higher accuracy and better robustness on the well‐known gait dataset CASIA‐B. Particularly in walking with coats cases, Gait‐TR gets a ∼90% accuracy rate. This result is higher than the best result of silhouette‐based models, which usually have higher accuracy than the skeleton‐based gait recognition models. Moreover, our experiment on CASIA‐B shows that the spatial transformer network can extract gait features from the human skeleton better than the widely used graph convolutional network.

With the improvement of face forgery techniques, people can easily generate various forgery face images, which brings serious challenges to public confidence. Numerous face forgery detection methods have been proposed, few of them can gain satisfying performance when training and testing forgery face images are generated by different forgery methods. To address this problem, we propose a domain-invariant representation learning (DIRL) method, which include feature distribution discrepancy minimization (FDDM) and optimal classier distance minimization (OCDM). FDDM adopts the idea of joint disentanglement and generative adversarial training to separate the irrelevant information and constrains the generator to reduce the discrepancy in feature distribution between source and target domains to obtain domain-shared features. OCDM uses the meta-learning method obtain a domain-invariant representation by minimizing the optimal classier distance between the domain-shared features of source and target domains. With FDDM and OCDM, we can learn an effective domain-invariant representation space in which there exists an ideal classifier equivalent to the optimal source domain classifier and the optimal target domain classifier, and the minimum classifier error can be achieved in both the source and target domains. Ablation experiments on the FaceForensics++ dataset show that compared to the baseline, our method can bring a decrease of 3.56% to 8.56% on Half Total Error Rate (HTER) when the test and training sets are generated by different face forgery methods. And the performance comparison experiments with the state-of-the-art methods also show that our method outperforms other methods.

Generative adversarial networks (GANs), as a leading class of generative models, have shown remarkable capacity in producing photo-realistic images. In practice, challenges remain in learning complex representations regarding data distribution. In this paper, we propose the structural pattern classification task, which enriches the training of GANs in a self-supervised manner. We first leverage the self-attention layer added to the generator to synthesize images with different structural patterns. Three distinct feature matrices are randomly swapped before calculating the self-attention feature maps. Each pattern stands for one possible combination. Meanwhile, we annotate real images with a fixed pattern. Then, the adversarial training is coupled with an auxiliary classification task. The discriminator needs to tell the correct structural pattern of input images. This auxiliary task provides an additional perspective for the discriminator to learn valuable representations of the data distribution. Empirical studies on CIFAR-10, STL-10, and CELEB-A demonstrate the effectiveness of our proposed structural pattern classification in improving the quality and diversity of the generated images, but limitations remain on corresponding interpretability research and exploration of images with higher resolution.

Cardiac arrhythmia is an abnormal rhythm of the heartbeat and can be life-threatening Electrocardiogram (ECG) is a technology that uses an electrocardiograph machine to record a graph of the changes in electrical activity produced by the heart at each cardiac cycle. ECG can generally be used to check whether the examinee has arrhythmia, ion channel disease, cardiomyopathy, electrolyte disorder and other diseases. To reduce the workload of doctors and improve the accuracy of ECG signal recognition, a novel and lightweight automatic ECG classification method based on Convolutional Neural Network (CNN) is proposed. The multi-branch network with different receptive fields is used to extract the multi-spatial deep features of heartbeats. The Channel Attention Module (CAM) and Bidirectional Long Short-Term Memory neural network (BLSTM) module are used to filter redundant ECG features. CAM and BLSTM are beneficial for distinguishing different categories of heartbeats. In the experiments, a four-fold cross-validation technique is used to improve the generalization capability of the network, and it shows good performance on the testing set. This method divides heartbeats into five categories according to the American Advancement of Medical Instrumentation (AAMI) criteria, which is validated in the MIT-BIH arrhythmia database. The sensitivity of this method to Ventricular Ectopic Beat (VEB) is 98.5% and the F1 score is 98.2%. The precision of the Supraventricular Ectopic Beat (SVEB) is 91.1%, and the corresponding F1 score is 90.8%. The proposed method has high classification performance and a lightweight feature. In a word, it has broad application prospects in clinical medicine and health testing.

Mitosis detection is one of the fundamental tasks in computational pathology, which is extremely challenging due to the heterogeneity of mitotic cell. Most of the current studies solve the heterogeneity in the technical aspect by increasing the model complexity. However, lacking consideration of the biological knowledge and the complex model design may lead to the overfitting problem while limited the generalizability of the detection model. In this paper, we systematically study the morphological appearances in different mitotic phases as well as the ambiguous non-mitotic cells and identify that balancing the data and feature diversity can achieve better generalizability. Based on this observation, we propose a novel generalizable framework (MitDet) for mitosis detection. The data diversity is considered by the proposed diversity-guided sample balancing (DGSB). And the feature diversity is preserved by inter- and intra- class feature diversity-preserved module (InCDP). Stain enhancement (SE) module is introduced to enhance the domain-relevant diversity of both data and features simultaneously. Extensive experiments have demonstrated that our proposed model outperforms all the SOTA approaches in several popular mitosis detection datasets in both internal and external test sets using minimal annotation efforts with point annotations only. Comprehensive ablation studies have also proven the effectiveness of the rethinking of data and feature diversity balancing. By analyzing the results quantitatively and qualitatively, we believe that our proposed model not only achieves SOTA performance but also might inspire the future studies in new perspectives. Source code is at https://github.com/Onehour0108/MitDet.

Identifying regulatory modules between miRNAs and genes is crucial in cancer research. It promotes a comprehensive understanding of the molecular mechanisms of cancer. The genomic data collected from subjects usually relate to different cancer statuses, such as different TNM Classifications of Malignant Tumors (TNM) or histological subtypes. Simple integrated analyses generally identify the core of the tumorigenesis (common modules) but miss the subtype-specific regulatory mechanisms (specific modules). In contrast, separate analyses can only report the differences and ignore important common modules. Therefore, there is an urgent need to develop a novel method to jointly analyze miRNA and gene data of different cancer statuses to identify common and specific modules. To that end, we developed a High-Order Graph Matching model to identify Common and Specific modules (HOGMCS) between miRNA and gene data of different cancer statuses. We first demonstrate the superiority of HOGMCS through a comparison with four state-of-the-art techniques using a set of simulated data. Then, we apply HOGMCS on stomach adenocarcinoma data with four TNM stages and two histological types, and breast invasive carcinoma data with four PAM50 subtypes. The experimental results demonstrate that HOGMCS can accurately extract common and subtype-specific miRNA-gene regulatory modules, where many identified miRNA-gene interactions have been confirmed in several public databases.