The Transformer-based methods provide a good opportunity for modeling the global context of gigapixel whole slide image (WSI), however, there are still two main problems in applying Transformer to WSI-based survival analysis task. First, the training data for survival analysis is limited, which makes the model prone to overfitting. This problem is even worse for Transformer-based models which require large-scale data to train. Second, WSI is of extremely high resolution (up to 150,000 x 150,000 pixels) and is typically organized as a multi-resolution pyramid. Vanilla Transformer cannot model the hierarchical structure of WSI (such as patch cluster-level relationships), which makes it incapable of learning hierarchical WSI representation. To address these problems, in this paper, we propose a novel Sparse and Hierarchical Transformer (SH-Transformer) for survival analysis. Specifically, we introduce sparse self-attention to alleviate the overfitting problem, and propose a hierarchical Transformer structure to learn the hierarchical WSI representation. Experimental results based on three WSI datasets show that the proposed framework outperforms the state-of-the-art methods.

Kirsten rat sarcoma viral oncogene homolog (namely KRAS) is a key biomarker for prognostic analysis and targeted therapy of colorectal cancer. Recently, the advancement of machine learning, especially deep learning, has greatly promoted the development of KRAS mutation detection from tumor phenotype data, such as pathology slides or radiology images. However, there are still two major problems in existing studies: inadequate single-modal feature learning and lack of multimodal phenotypic feature fusion. In this paper, we propose a Disentangled Representation-based Multimodal Fusion framework integrating Pathomics and Radiomics (DRMF-PaRa) for KRAS mutation detection. Specifically, the DRMF-PaRa model consists of three parts: (1) the pathomics learning module, which introduces a tissue-guided Transformer model to extract more comprehensive and targeted pathological features; (2) the radiomics learning module, which captures the generic hand-crafted radiomics features and the task-specific deep radiomics features; (3) the disentangled representation-based multimodal fusion module, which learns factorized subspaces for each modality and provides a holistic view of the two heterogeneous phenotypic features. The proposed model is developed and evaluated on a multi modality dataset of 111 colorectal cancer patients with whole slide images and contrast-enhanced CT. The experimental results demonstrate the superiority of the proposed DRMF-PaRa model with an accuracy of 0.876 and an AUC of 0.865 for KRAS mutation detection.

Survival analysis is a significant study in cancer prognosis, and the multi-modal data, including histopathological images, genomic data, and clinical information, provides unprecedented opportunities for its development. However, because of the high dimensionality and the heterogeneity of histopathological images and genomic data, acquiring effective predictive characters from these multi-modal data has always been a challenge for survival analysis. In this article, we propose a transformer-based survival analysis model (TransSurv) for colorectal cancer that can effectively integrate intra-modality and inter-modality features of histopathological images, genomic data, and clinical information. Specifically, to integrate the intra-modality relationship of image patches, we develop a multi-scale histopathological features fusion transformer (MS-Trans). Furthermore, we provide a cross-modal fusion transformer based on cross attention for multi-scale pathological representation and multi-omics representation, which includes RNA-seq expression and copy number alteration (CNA). At the output layer of the TransSurv, we adopt the Cox layer to integrate multi-modal fusion representation with clinical information for end-to-end survival analysis. The experimental results on the Cancer Genome Atlas (TCGA) colorectal cancer cohort demonstrate that the proposed TransSurv outperforms the existing methods and improves the prognosis prediction of colorectal cancer.

ABSTRACT Background Immunoglobulin A nephropathy (IgAN) and idiopathic membranous nephropathy (IMN) are the most common glomerular diseases. Immunofluorescence (IF) tests of renal tissues are crucial for the diagnosis. We developed a multiple convolutional neural network (CNN)-facilitated diagnostic program to assist the IF diagnosis of IgAN and IMN. Methods The diagnostic program consisted of four parts: a CNN trained as a glomeruli detection module, an IF intensity comparator, dual-CNN (D-CNN) trained as a deposition appearance and location classifier and a post-processing module. A total of 1573 glomerular IF images from 1009 patients with glomerular diseases were used for the training and validation of the diagnostic program. A total of 1610 images of 426 patients from different hospitals were used as test datasets. The performance of the diagnostic program was compared with nephropathologists. Results In >90% of the tested images, the glomerulus location module achieved an intersection over union >0.8. The accuracy of the D-CNN in recognizing irregular granular mesangial deposition and fine granular deposition along the glomerular basement membrane was 96.1% and 93.3%, respectively. As for the diagnostic program, the accuracy, sensitivity and specificity of diagnosing suspected IgAN were 97.6%, 94.4% and 96.0%, respectively. The accuracy, sensitivity and specificity of diagnosing suspected IMN were 91.7%, 88.9% and 95.8%, respectively. The corresponding areas under the curve (AUCs) were 0.983 and 0.935. When tested with images from the outside hospital, the diagnostic program showed stable performance. The AUCs for diagnosing suspected IgAN and IMN were 0.972 and 0.948, respectively. Compared with inexperienced nephropathologists, the program showed better performance. Conclusion The proposed diagnostic program could assist the IF diagnosis of IgAN and IMN.

随着环保要求的日趋严格,对动车组设计提出了更高的要求.由于混合动力动车组能够有效降低排放量,合理利用能源并提高车辆运行效率,契合了低碳环保的主题,因此混合动力动车组成为了发展的新趋势.能量管理策略作为柴电双模混合动力动车组设计的关键控制点,对于项目的成败具有决定性作用.对柴电双模混合动力动车组的能量管理策略进行研究,从系统构成、能量流动特性和功率分配等方面详细阐述柴电双模混合动力动车组能量管理策略.

为确保铁道客车符合欧盟铁路互联互通性技术规范对残疾和行动不便人士(PRM)无障碍乘坐车辆的技术规定,设计欧洲铁道客车时,应根据线路条件和车辆参数,提前规划PRM乘客乘车路径.根据站台尺寸和车辆参数,计算车门相对站台的距离.根据PRM乘客乘车相关的技术标准和法律法规,以及设计欧洲铁道客车的经验,对车门、台阶、乘车辅助设施开展分析,并给出确定车门参数、台阶数量和尺寸、扶手位置的方法,以及乘车辅助设施使用情形.

目前在国内外还有相当存量的非电气化线路,因此内燃动力机车或者采用动力包的混合动力列车在相当长的时间内会一直存在.动力包作为该种列车的动力来源是其关键子系统,在车辆设计时必须充分考虑并进行正确选型.对动力包选型依照设计时需考虑的多种因素展开分析研究,提出了动力包型谱,为内燃动力机车或混合动力列车的动力包选型提供参考.

为了更加全面地了解和掌握目前国内外相关的PRM设计要求,主要介绍了欧洲轨道交通领域针对无障碍通过能力的相关技术要求及规定,根据执行的多个欧洲项目经验梳理总结出了主要的设计注意项点.对轨道交通车辆当前典型设计形式进行了分析总结,识别出适用于行动不便人士的车厢和普通车厢设计的差异性,进而为年轻设计师提供正确的设计思路,节约宝贵的设计资源,降低企业生产认证成本,提升企业市场竞争力.

某出口动车组在线路运行调试过程中多次出现动力包自动停机现象,停机时柴油机报出燃油压力低故障.针对此问题,首先进行了前期故障排查与分析,找出故障发生的原因并提出了解决措施,最终通过试验运行验证了优化方案的有效性.

Microsatellite instability (MSI) is a crucial biomarker to clinical immunotherapy in gastrointestinal cancer, while additional immunohistochemical or genetic tests for MSI are generally missing due to lack of medical resources. Deep learning has achieved promising performance in detecting MSI from hematoxylin and eosin (H &E) stained histopathology slides. However, these methods are primarily based on patch-supervised slide-label models and then aggregate patch-level results into the slides-level result, resulting unstable prediction due to noisy patches and aggregation ways. In this paper, we propose a joint region-attention and multi-scale transformer (RAMST) network for microsatellite instability detection from whole slide images in gastrointestinal cancer. Specifically, we present a region-attention mechanism and a feature weight uniform sampling (FWUS) method to learn a representative subset of image patches from whole slide images. Moreover, we introduce the transformer architecture to fuse the multi-scale histopathology features consisting of patch-level features with region-level features to characterize the whole slide images for slide-level MSI detection. Compared to the existing MSI detection methods, the proposed RAMST shows the best performances on the colorectal and stomach cancer dataset from The Cancer Genome Atlas (TCGA) and provides an effective features representation learning method for WSI-label tasks.

介绍了一种基于柴油机内燃动力包和超级电容的混合动力系统的构成和控制方法,就该混合动力系统能量管理策略、动力模式控制方法等进行了说明.通过实验,对各个工况进行了验证,试验结果证实了该混合动力系统及其控制方法的有效性.

主要介绍了某出口动车组转向架区域干涉分析方法,提出了对转向架区域底架线槽、制动管路、卫生间管路布置要求,并对试验方法及评判标准作出了规定.

Survival analysis is crucial to the evaluation of cancer treatment options and deep learning-based methods integrating pathological images and genomic data have been used for prognosis prediction. However, the most methods are based on the analysis of pathological image patches, thus ignoring the morphological structure information at larger field-of-view and intrinsic relationships between patches. Meanwhile, the existing models fail to exploit the powerful representation learning capabilities of the neural networks for effective multimodal feature fusion of pathological images and genomic data. In this paper, we propose a novel transformer-based fusion network integrating pathological images and genomic data (PGTFNet) for cancer survival analysis. Specifically, we present a transformer-based feature fusion module for multi-scale pathological slides to fully exploit the intra-modality relationships between image patches at various fields of view. Moreover, in order to make effective inter-modality feature fusion of pathological images and genomic data, we introduce a cross-attention transformer module that can exchange feature representations of different modalities between two transformers branches. The PG-TFNet is performed on the colorectal cancer dataset from the Cancer Genome Atlas (TCGA), which contains paired whole-slide images and genomic data with ground truth survival data. The experimental results from a 10-fold cross validation demonstrate that the proposed PG-TFNet facilitates the prognosis prediction of colorectal cancer and shows superiority over the existing methods.

BACKGROUND:Deep learning algorithms significantly improve the accuracy of pathological image classification, but the accuracy of breast cancer classification using only single-mode pathological images still cannot meet the needs of clinical practice. Inspired by the real scenario of pathologists reading pathological images for diagnosis, we integrate pathological images and structured data extracted from clinical electronic medical record (EMR) to further improve the accuracy of breast cancer classification.METHODS:In this paper, we propose a new richer fusion network for the classification of benign and malignant breast cancer based on multimodal data. To make pathological image can be integrated more sufficient with structured EMR data, we proposed a method to extract richer multilevel feature representation of the pathological image from multiple convolutional layers. Meanwhile, to minimize the information loss for each modality before data fusion, we use the denoising autoencoder as a way to increase the low-dimensional structured EMR data to high-dimensional, instead of reducing the high-dimensional image data to low-dimensional before data fusion. In addition, denoising autoencoder naturally generalizes our method to make the accurate prediction with partially missing structured EMR data.RESULTS:The experimental results show that the proposed method is superior to the most advanced method in terms of the average classification accuracy (92.9%). In addition, we have released a dataset containing structured data from 185 patients that were extracted from EMR and 3764 paired pathological images of breast cancer, which can be publicly downloaded from http://ear.ict.ac.cn/?page_id=1663 .CONCLUSIONS:We utilized a new richer fusion network to integrate highly heterogeneous data to leverage the structured EMR data to improve the accuracy of pathological image classification. Therefore, the application of automatic breast cancer classification algorithms in clinical practice becomes possible. Due to the generality of the proposed fusion method, it can be straightforwardly extended to the fusion of other structured data and unstructured data.

Building upon the clinical evidence supporting that decomposing a pathological image into different components can improve diagnostic value, in this paper we propose a Decomposition-and-Fusion Network (DFNet) for HE-stained pathological image classification. The medical goal of using HE-stained pathological images is to distinguish between nucleus, cytoplasm and extracellular matrix, thereby displaying the overall layouts of cells and tissues. We embed this most basic medical knowledge into a deep learning framework that decomposes a pathological image into cell nuclei and the remaining structures (that is, cytoplasm and extracellular matrix). With such decomposed pathological images, DFNet first extracts independent features using three independent CNN branches, and then gradually merges these features together for final classification. In this way, DFNet is able to learn more representative features with respect to different structures and hence improve the classification performance. Experimental results on two different datasets with various cancer types show that the DFNet achieves competitive performance.

Automatic breast cancer grading methods based on HE stained pathological images can be summarized into two categories. The first category is to use learning-based methods to directly extract the features of the pathological image for breast cancer grading. However, unlike the coarse-grained problem of breast cancer classification, grading of breast Invasive Ductal Carcinoma (IDC) is a fine-grained classification problem. Only using general methods cannot classify IDC well. The second category is to conduct the three evaluation criteria of Nottingham Grading System (NGS) separately, and then integrate the results of the three criteria to obtain the final IDC grading result. However, NGS is only a semi-quantitative evaluation method. The inherent medical motivation of NGS is to grade IDC with the help of nuclei-related features. In this paper, we proposed a nuclei-aware network for IDC grading in pathological images. The entire network achieves an effect similar to the attention mechanism in end-to-end learning, so as to learn fine-grained and nuclei-related feature representations for IDC grading. It should to be pointed out that our method can emphasize custom areas, thus providing a way to model medical knowledge into the network structure. This is different from the general attention mechanism that cannot artificially control the area of attention. Experimental results show that the performance of proposed method is better than the state-of-the-art.

A complete and detailed cerebrovascular image segmented from time-of-flight magnetic resonance angiography (TOF-MRA) data is essential for the diagnosis and therapy of the cerebrovascular diseases. In recent years, three-dimensional cerebrovascular segmentation algorithms based on statistical models have been widely used, but the existed methods always perform poorly on stenotic vessels and are not robust enough. In this paper, we propose a parallel cerebrovascular segmentation algorithm based on focused multi-Gaussians model and heterogeneous Markov random field. Specifically, we present a focused multi-Gaussians (FMG) model with local fitting region to model the vascular tissue more accurately and introduce the chaotic oscillation particle swarm optimization (CO-PSO) algorithm to improve the global optimization capability in the parameter estimation. Furthermore, we design a heterogeneous Markov Random Field (MRF) in the three-dimensional neighborhood system to incorporate precise local character of image. Finally, the algorithm has been performed parallel optimization based on GPUs and obtain about 60 times speedup compared to serial execution. The experiments show that the proposed algorithm can produce more detailed segmentation result in shorter time and performs well on the stenotic vessels robustly.

Breast cancer grading methods based on hematoxylin-eosin (HE) stained pathological images can be summarized into two categories. The first category is to directly extract the pathological image features for breast cancer grading. However, unlike the coarse-grained problem of breast cancer classification, breast cancer grading is a fine-grained classification problem, so general methods cannot achieve satisfactory results. The second category is to apply the three evaluation criteria of the Nottingham Grading System (NGS) separately, and then integrate the results of the three criteria to obtain the final grading result. However, NGS is only a semiquantitative evaluation method, and there may be far more image features related to breast cancer grading. In this paper, we proposed a Nuclei-Guided Network (NGNet) for breast invasive ductal carcinoma (IDC) grading in pathological images. The proposed nuclei-guided attention module plays the role of nucleus attention, so as to learn more nuclei-related feature representations for breast IDC grading. In addition, the proposed nuclei-guided fusion module in the fusion process of different branches can further enable the network to focus on learning nuclei-related features. Overall, under the guidance of nuclei-related features, the entire NGNet can learn more fine-grained features for breast IDC grading. The experimental results show that the performance of the proposed method is better than that of state-of-the-art method. In addition, we released a well-labeled dataset with 3644 pathological images for breast IDC grading. This dataset is currently the largest publicly available breast IDC grading dataset and can serve as a benchmark to facilitate a broader study of breast IDC grading.

Super-resolution nanoscopy based on wide-field microscopic imaging provided high efficiency but limited resolution. Here, we demonstrate a general strategy to push its resolution down to ~50 nm, which is close to the range of single molecular localization microscopy, without sacrificing the wide-field imaging advantage. It is done by actively and simultaneously modulating the characteristic emission of each individual emitter at high density. This method is based on the principle of excited state coherent control on single-particle two-photon fluorescence. In addition, the modulation efficiently suppresses the noise for imaging. The capability of the method is verified both in simulation and in experiments on ZnCdS quantum dot-labeled films and COS7 cells. The principle of coherent control is generally applicable to single-multiphoton imaging and various probes.

Abstract Background and Aims Immunofluorescence (IF) tests of renal tissue are of great value in diagnosing most of the glomerular diseases. However, high quality IF tests and results interpretation by experienced pathologists are not universally available in different areas of China. The development of deep neural networks has been used to facilitate digital analysis of pathologic images recently. We proposed a novel Convolutional Residual Dense Network (CR-DenseNet) to facilitate IF assessment of renal biopsy samples. Method A dataset with 725 IF images, including 312 images of IgA nephropathy (IgAN), 319 images of idiopathic membranous nephropathy (IMN) and 94 images of type V lupus nephritis (LN V) diagnosed in Peking Union Medical College Hospital (PUMCH) from November, 2016 to March, 2018 were used for training and validation of CR-DenseNet. All the images were photographed using ANDOR, ZYLA, Japan. The resolution was 2560 × 2160 pixels. These images were carefully annotated for the distribution characteristics and final diagnosis by two renal pathologists independently. IgAN showed mesangial predominating deposition. IMN showed glomerular basement membrane (GBM) predominating deposition. LN V showed positive depositions in both mesangial area and GBM. These three groups were used for IF distribution identification training. In CR-DenseNet, convolutional residual dense blocks were introduced, each of them consisted of a dense block with a convolutional skip connection to fully exploit the dense local features. To identify the deposition location, we imposed a switch in the proposed model to handle an additional input for different types of tasks, which can provide glomerular contours approximated from the IF image foreground (Figure 1). Performance was evaluated using overall accuracy and F1 score. F1 was computed as 2×True Positive / (2×True Positive + False Positive + False Negative). The whole protocol was approved by Institutional Review Board of PUMCH (No. S-K913). Results Experimental results showed that the proposed CR-DenseNet model outperformed the state-of-the-art method. In identification of segmental and glomerular IF positive images, our model showed overall accuracy of 92.0%. The true positive rate and F1 score of segmental positive samples recognition were 90.1% and 0.889. The true positive rate and F1 score of glomerular recognition positive samples were 94.1% and 0.938. Furthermore, the overall accuracy of our model in identifying mesangial predominating deposition (IgAN), GBM predominating deposition (IMN) as well as positive depositions in both mesangial area and GBM (LN V) was 91.2%. The true positive rates of each above-mentioned deposition classification were 93.1%, 90.5% and 88.9%, respectively. The corresponding F1 scores were 0.964, 0.905 and 0.889. Conclusion Our preliminary data showed that CR-DenseNet model was quite powerful in making IF diagnosis of typical glomerular diseases for the first time.