The Deepwater Horizon (DWH) oil spill in the NE Gulf of Mexico in 2010 was the largest accidental spill globally. During and after the spill, oil was incorporated into marine snow (marine-oil-snow or MOS) and sedimented to the seafloor, accounting for similar to 21% of the oil released. We assessed the abundance, distribution, size, composition, shape, and fractal dimension of marine snow in relation to the oil spill and natural environmental conditions between May 2010 and August 2014 using SIPPER camera images. The Mississippi River outflow, which governs productivity in this region, was elevated during the oil spill resulting in a large phytoplankton bloom. Integrated marine snow abundance was significantly higher during the spill and particle size spectra showed a higher abundance of intermediate- and large-sized particles during the oil spill compared to non-spill years. Particle collision and sinking rates may be sensitive to particle composition, shape, and fractal dimension (surface area complexity). Identifiable aggregate components included chain diatoms and acantharian radiolarians and their spines. Almost all aggregates were elongated in all years. The geometric mean fractal dimension for particles in non-oil spill years was 1.45 (range: 1.00-1.94), whereas particle fractal dimensions were significantly higher (mean: 1.49, 1.01-1.87, p = 0.03) during the oil spill, indicating that oil droplets in MOS may have decreased particle porosity compared to other years. Fractal dimensions also increased with particle size during all years. These data provide a baseline for model development, responding to future oil spills, and for investigations of biogeochemical cycles in this region.

Cytomorphological analysis of the bone marrow aspirate (BMA) is pivotal for the diagnostic workup of a broad range of hematological disorders. However, this skill is error prone, highly complex, and time consuming. Deep learning–based models for the automatic classification of bone marrow cell morphology demonstrate the potential to improve diagnostic efficiency and accuracy. However, existing deep learning approaches in this field fall short of expert-level performance and lack generalizability beyond a single dataset. Working with multiple hematopathologists, we curated a dataset from the University of California, San Francisco, which included a training set of 30,394 images from 40 patients with morphologically normal marrows and a test set of 8507 images from 10 different patients, all derived from 400×-equivalent whole-slide images (WSIs). We then developed DeepHeme, a snapshot ensemble deep learning classifier, which outperformed previous models in accuracy while expanding the total number of differentiable cell classes. We externally validated DeepHeme using an independent dataset from the Memorial Sloan Kettering Cancer Center, which included 2694 images from 10 morphologically normal patients and 11,076 images from 655 patients with normal or diseased marrow, scanned using a different WSI system, demonstrating robust generalizability. At the level of individual cell classifications, we systematically compared DeepHeme’s diagnostic performance with that of three medical experts from different academic hospitals, demonstrating that DeepHeme achieved accuracy comparable to, or exceeding, that of human experts. Accurate and generalizable cell classification represents a step toward automated analysis of hematopathology slides and the development of quantitative, morphology-based, predictive markers.

Advancements in deep neural network-based models (deep AI) can be used to provide clinical assessments of Gleason patterns in digital prostate pathology slides. We cover the variability in clinical assessments which provides challenges in training a deep neural model with small glandular regions with limited sample sizes. Method: A convolutional neural network (CNN) model with transfer learning was used to train on pathology images with smaller patches that provide representation of a section with multiple glands of similar glandular architecture (primary Gleason pattern) level data (300x300 pixel or 40-140 micron patches). We assembled a cohort of Hematoxylin Eosin (H&E) stained sections with patches (3311 GS3 and 2909 GS4 patches) extracted from whole slide images from a cohort of 58 patients. We used calibrated confidence on the clinical scoring of the pathological slides at the pre-training phase. We conducted seven experiments splitting the data randomly, but with the constraint that all patches in holdout sets were not part of training (at patient biopsy whole slide level). For each experiment the training data was further split into 5 training-validation folds, that results in 35 models for each ablation study. We compare our findings to a naïve baseline. The baseline used in this study comprises: a VGG16 network with all CNN layers (5 stage) but with a smaller 2-stage fully-connected output classification layer (as implemented for all models), trained with an AdamW optimizer (with default TensorFlow settings), original ground-truth labels (no label flipping or removal of training samples), and no confidence filtering. Additionally, the optimal case (averaged over 5 folds) is compared to unoptimized case without ensembling. In both the unoptimized and optimized cases, all models were trained for ∼600 epochs, and the models with lowest validation loss were saved. We found optimized AI models were able to improve classification of the primary score at the patch level (Gleason 3 vs 4) using ensembles, training-label reassignment and ambiguous sample elimination, and most dramatically by filtering on the highest confidence samples. We achieved a mean accuracy of 0.74, F 1 of 0.72 and AUC of 0.79 estimated using holdout sets classifying small patches with Gleason 3 and 4 patterns. Average performance on all metrics were improved by roughly 20%, when accounting for all optimizations Ryan Fogarty, Dmitry Goldgof, Laerence Hall, Jasreman Dhillon, Vaibhav Chumbalkar, Yoganand Balagurunathan. Confident Filtering in the context of primary Gleason pattern classification [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Artificial Intelligence and Machine Learning; 2025 Jul 10-12; Montreal, QC, Canada. Philadelphia (PA): AACR; Clin Cancer Res 2025;31(13_Suppl):Abstract nr B029.

Background: The need for sharing large-scale datasets in training deep learning models, particularly in healthcare, raises significant data security and privacy concerns. To address these issues, methods such as data encryption or encoding are utilized. These techniques can encrypt the data and make it unreadable to humans, while still retaining its usefulness for training models.Method: In this study, we investigate various image encoding techniques designed to protect privacy by making images unrecognizable while still retaining their usefulness for model training. Our investigation utilized a publicly available facial database and focused on evaluating the trade-offs inherent in image encoding techniques, with a special emphasis on balancing privacy and model accuracy.Conclusion: This study navigate the balance between protecting sensitive data and meeting the data demands necessary for effective model training. It sheds light on the intricate trade-offs among different image encoding techniques and offers insights into finding an optimal balance between privacy protection and model performance.

Large language models (LLM) are perceived to offer promising potentials for automating security tasks, such as those found in security operation centers (SOCs). As a first step towards evaluating this perceived potential, we investigate the use of LLMs in software pentesting, where the main task is to automatically identify software security vulnerabilities in source code. We hypothesize that an LLM-based AI agent can be improved over time for a specific security task as human operators interact with it. Such improvement can be made, as a first step, by engineering prompts fed to the LLM based on the responses produced, to include relevant contexts and structures so that the model provides more accurate results. Such engineering efforts become sustainable if the prompts that are engineered to produce better results on current tasks, also produce better results on future unknown tasks. To examine this hypothesis, we utilize the OWASP Benchmark Project 1.2 which contains 2,740 hand-crafted source code test cases containing various types of vulnerabilities. We divide the test cases into training and testing data, where we engineer the prompts based on the training data (only), and evaluate the final system on the testing data. We compare the AI agent's performance on the testing data against the performance of the agent without the prompt engineering. We also compare the AI agent's results against those from SonarQube, a widely used static code analyzer for security testing. We built and tested multiple versions of the AI agent using different off-the-shelf LLMs -- Google's Gemini-pro, as well as OpenAI's GPT-3.5-Turbo and GPT-4-Turbo (with both chat completion and assistant APIs). The results show that using LLMs is a viable approach to build an AI agent for software pentesting that can improve through repeated use and prompt engineering.

The progress of incorporating deep learning in the field of medical image interpretation has been greatly hindered due to the tremendous cost and time associated with generating ground truth for supervised machine learning, alongside concerns about the inconsistent quality of images acquired. Active learning offers a potential solution to these problems of expanding dataset ground truth by algorithmically choosing the most informative samples for ground truth labeling. Still, this effort incurs the costs of human labeling, which needs minimization. Furthermore, automatic labeling approaches employing active learning often exhibit overfitting tendencies while selecting samples closely aligned with the training set distribution and excluding out-of-distribution samples, which could potentially improve the model’s effectiveness. We propose that the majority of out-of-distribution instances can be attributed to inconsistent cross images. Since the FDA approved the first whole-slide image system for medical diagnosis in 2017, whole-slide images have provided enriched critical information to advance the field of automated histopathology. Here, we exemplify the benefits of a novel deep learning strategy that utilizes high-resolution whole-slide microscopic images. We quantitatively assess and visually highlight the inconsistencies within the whole-slide image dataset employed in this study. Accordingly, we introduce a deep learning-based preprocessing algorithm designed to normalize unknown samples to the training set distribution, effectively mitigating the overfitting issue. Consequently, our approach significantly increases the amount of automatic region-of-interest ground truth labeling on high-resolution whole-slide images using active deep learning. We accept 92% of the automatic labels generated for our unlabeled data cohort, expanding the labeled dataset by 845%. Additionally, we demonstrate expert time savings of 96% relative to manual expert ground-truth labeling.

Deep Learning (DL)-based solutions have shown promising performance in assessing neonatal pain. However, the occlusion of the visual modality (face and body) is common in clinical settings due to several factors, including a prone sleeping position, low light, or swaddling. In such scenarios, other pain signals, such as audio signals, can be used as the major behavioral signs of pain. Although DL-based methods are proposed to assess pain from audio, these methods lack transparency and explainability (black box), which can decrease the user's trust in the automated decision. In this work, we visualize the neonate's audio signal as a spectrogram image to classify it as pain or no pain and present an instance-based approach for explaining the decision of the black-box model. Further, this work provides an analysis of the most helpful and harmful training instances using an influence score followed by assessing their impact on pain prediction. Experimental results demonstrate that the proposed approach can detect and remove harmful instances, eventually leading to a compressed dataset. Our results also show that the proposed work can add explainability to the current DL-based pain detection methods, which can enhance users' trust and provide a viable approach toward pain assessment in clinical settings.

Morphology-based classification of cells in the bone marrow aspirate (BMA) is a key step in the diagnosis and management of hematologic malignancies. However, it is time-intensive and must be performed by expert hematopathologists and laboratory professionals. We curated a large, high-quality dataset of 41,595 hematopathologist consensus-annotated single-cell images extracted from BMA whole slide images (WSIs) containing 23 morphologic classes from the clinical archives of the University of California, San Francisco. We trained a convolutional neural network, DeepHeme, to classify images in this dataset, achieving a mean area under the curve (AUC) of 0.99. DeepHeme was then externally validated on WSIs from Memorial Sloan Kettering Cancer Center, with a similar AUC of 0.98, demonstrating robust generalization. When compared to individual hematopathologists from three different top academic medical centers, the algorithm outperformed all three. Finally, DeepHeme reliably identified cell states such as mitosis, paving the way for image-based quantification of mitotic index in a cell-specific manner, which may have important clinical applications.

Across basic research studies, cell counting requires significant human time and expertise. Trained experts use thin focal plane scanning to count (click) cells in stained biological tissue. This computer-assisted process (optical disector) requires a well-trained human to select a unique best z-plane of focus for counting cells of interest. Though accurate, this approach typically requires an hour per case and is prone to inter- and intra-rater errors. Our group has previously proposed deep learning (DL)-based methods to automate these counts using cell segmentation at high magnification. Here we propose a novel You Only Look Once (YOLO) model that performs cell detection on multi-channel z-plane images (disector stack). This automated Multiple Input Multiple Output (MIMO) version of the optical disector method uses an entire z-stack of microscopy images as its input, and outputs cell detections (counts) with a bounding box of each cell and class corresponding to the z-plane where the cell appears in best focus. Compared to the previous segmentation methods, the proposed method does not require time- and labor-intensive ground truth segmentation masks for training, while producing comparable accuracy to current segmentation-based automatic counts. The MIMO-YOLO method was evaluated on systematic-random samples of NeuN-stained tissue sections through the neocortex of mouse brains (n=7). Using a cross validation scheme, this method showed the ability to correctly count total neuron numbers with accuracy close to human experts and with 100% repeatability (Test-Retest).

Current face detection algorithms are extremely generalized and can obtain decent accuracy when detecting the adult faces. These approaches are insufficient when handling outlier cases, for example when trying to detect the face of a neonate infant whose face composition and expressions are relatively different than that of the adult. It is furthermore difficult when applied to detect faces in a complicated setting such as the Neonate Intensive Care Unit. By training a state-of-the-art face detection model, You-Only-Look-Once, on a proprietary dataset containing labelled neonate faces in a clinical setting, this work achieves near real time neonate face detection. Our preliminary findings show an accuracy of 68.7%, compared to the off the shelf solution which detected neonate faces with an accuracy of 7.37%. Although further experiments are needed to validate our model, our results are promising and prove the feasibility of detecting neonatal faces in challenging real-world settings. The robust and real-time detection of neonatal faces would benefit wide range of automated systems (e.g., pain recognition and surveillance) who currently suffer from the time and effort due to the necessity of manual annotations. To benefit the research community, we make our trained weights publicly available at github(https://github.com/ja05haus/trained_neonate_face).

When a learned model has high accuracy under familiar settings (internal testing) and a big drop in accuracy under slightly different circumstances (external testing) we suspect it is using shortcuts to make decisions. This problem is known as shortcut learning. In medical imaging, shortcuts are undesired and unintended features that the model relies on to perform diagnosis. Shortcut-based solutions using medical images could lead to false diagnoses and have dangerous implications for patients. In the current COVID-19 era, a large set of papers have been published proposing the use of deep convolutional neural networks to perform diagnosis or triage of COVID-19 from chest X-rays (CXRs). These studies are reporting high accuracies which could be misleading and overestimated. To our knowledge, none of the currently published papers with high performance reported testing on samples from truly unseen data sources. Studies which did, have noticed a significant performance drop when testing on unseen sources indicating a failure to generalize. In this paper, we elucidate the generalization challenge of deep learning based models trained for disease diagnosis. We use the example of COVID-19 diagnosis from CXRs. Solutions that mitigate shortcut learning are introduced and experimentally shown to be effective. Our proposed methods enable the models to have a statistically significantly reduced performance drop-off on unseen data sources. Thus, lowering the performance drop to only 9% instead of 20%. The issues with convolutional neural networks addressed here generally apply to other imaging modalities and recognition problems, as shown.

Lung cancer is the leading cause of cancer-related deaths responsible for over 130,000 deaths each year in the US. Early diagnosis and prompt treatment is crucial for prolonging survival. Many studies have been performed with regard to detecting lung cancer using computed tomography images, or classifying the type of a cancer from pathology images. Some research focuses on survival time prediction instead, which consists of ranking patients according to their expected survival time. This has proven to be a difficult task, and most approaches offer only a slight improvement over random guessing. Instead of ranking patients according to survival time, we propose to predict whether a patient falls into a long- or short-term survival group. We show that this approach outperforms regression-based approaches when predicting precise survival time is not necessary. In addition to that, we show that it is possible to predict short-term and long-term survival from lung cancer histopathology images without ROI annotations. We have obtained an 0.81 AUC when predicting whether a patient would fall into the short-term survival group (less than 12 months) or long-term survival group (greater than 60 months). Furthermore, we show that our model is capable of classifying patients into long and short-term survival even when their survival time falls outside of our chosen ranges.

Glioma is the most common type of primary malignant brain tumor. Accurate survival time prediction for glioma patients may positively impact treatment planning. In this paper, we develop an automatic survival time prediction tool for glioblastoma patients along with an effective solution to the limited availability of annotated medical imaging datasets. Ensembles of snapshots of three dimensional (3D) deep convolutional neural networks (CNN) are applied to Magnetic Resonance Image (MRI) data to predict survival time of high-grade glioma patients. Additionally, multi-sequence MRI images were used to enhance survival prediction performance. A novel way to leverage the potential of ensembles to overcome the limitation of labeled medical image availability is shown. This new classification method separates glioblastoma patients into long- and short-term survivors. The BraTS (Brain Tumor Image Segmentation) 2019 training dataset was used in this work. Each patient case consisted of three MRI sequences (T1CE, T2, and FLAIR). Our training set contained 163 cases while the test set included 46 cases. The best known prediction accuracy of 74% for this type of problem was achieved on the unseen test set.

Artificial Intelligence (AI)-based methods allow for automatic assessment of pain intensity based on continuous monitoring and processing of subtle changes in sensory signals, including facial expression, body movements, and crying frequency. Currently, there is a large and growing need for expanding current AI-based approaches to the assessment of postoperative pain in the neonatal intensive care unit (NICU). In contrast to acute procedural pain in the clinic, the NICU has neonates emerging from postoperative sedation, usually intubated, and with variable energy reserves for manifesting forceful pain responses. Here, we present a novel multi-modal approach designed, developed, and validated for assessment of neonatal postoperative pain in the challenging NICU setting. Our approach includes a robust network capable of efficient reconstruction of missing modalities (e.g., obscured facial expression due to intubation) using an unsupervised spatio-temporal feature learning with a generative model for learning the joint features. Our approach generates the final pain score along with the intensity using an attentional cross-modal feature fusion. Using experimental dataset from postoperative neonates in the NICU, our pain assessment approach achieves superior performance (AUC 0.906, accuracy 0.820) as compared to the state-of-the-art approaches.

Lung cancer has high mortality and occurrence worldwide. Radiomics is a method for extracting quantitative features from medical images that can be used for predictive analysis. Radiomics has been applied quite successfully for lung nodule malignancy prediction. Along with traditional radiomics, Convolutional Neural Networks (CNN) are now used quite effectively for lung cancer analysis. Texture provides information about variation in pixel intensity in regions. Lung nodules/tumors possess a noticeable texture pattern. That’s why texture radiomics features can be used to construct predictive models to analyze malignant and benign lung nodules. As textures show visible patterns, training the CNNs using texture images is a novel idea that enables the creation of an ensemble of classifiers. In this study, 192 texture images (wavelet, Laws, gray level zone matrix, neighborhood grey tone difference, and run-length) were generated, and the same CNN architecture was trained separately on all texture images. We termed this approach, “Deep Radiomics.” The maximum classification accuracy of 73% and 0.82 AUC was achieved from both the P2L2C5 wavelet and L5E5L5 laws texture images. When multiple CNN model’s predictions were merged to generate an ensemble model, results of 81.43% (0.91 AUC) were achieved from our study, which was an improvement.

Ram Adapa (Fellow, IEEE) received the B.S. degree from Jawaharlal Nehru Technological University, India, the M.S. degree from IIT Kanpur, India, and the Ph.D. degree from the University of Waterloo, ON, Canada, all in electrical engineering.

Identifying who is infected with the Covid-19 virus is critical for controlling its spread. X-ray machines are widely available worldwide and can quickly provide images that can be used for diagnosis. A number of recent studies claim it may be possible to build highly accurate models, using deep learning, to detect Covid-19 from chest X-ray images. This paper explores the robustness and generalization ability of convolutional neural network models in diagnosing Covid-19 disease from frontal-view (AP/PA), raw chest X-ray images that were lung field cropped. Some concerning observations are made about high performing models that have learned to rely on confounding features related to the data source, rather than the patient's lung pathology, when differentiating between Covid-19 positive and negative labels. Specifically, these models likely made diagnoses based on confounding factors such as patient age or image processing artifacts, rather than medically relevant information.

This paper presents the first multimodal neonatal pain dataset that contains visual, vocal, and physiological responses following clinically required procedural and postoperative painful procedures. It was collected from 58 neonates (27-41 gestational age) during their hospitalization in the neonatal intensive care unit. The visual and vocal data were recorded using an inexpensive RGB camera while the physiological responses (vital signs and cortical activity) were recorded using portable bedside monitors. The recorded behavioral and physiological responses were scored by expert nurses using two validated pain scales to obtain the ground truth labels. In addition to behavioral and physiological responses, our dataset contains clinical information such as the neonate’s age, gender, weight, pharmacological and non-pharmacological interventions, and previous painful procedures. The presented multimodal dataset can be used to develop artificial intelligence systems that monitor, assess, and predict neonatal pain based on the analysis of behavioral and physiological responses. It can also be used to advance the understanding of neonatal pain, which can lead to the development of effective pain prevention and treatment.

Lung cancer is by far the leading cause of cancer death in the US. Recent studies have demonstrated the effectiveness of screening using low dose CT (LDCT) in reducing lung cancer related mortality. While lung nodules are detected with a high rate of sensitivity, this exam has a low specificity rate and it is still difficult to separate benign and malignant lesions. The ISBI 2018 Lung Nodule Malignancy Prediction Challenge, developed by a team from the Quantitative Imaging Network of the National Cancer Institute, was focused on the prediction of lung nodule malignancy from two sequential LDCT screening exams using automated (non-manual) algorithms. We curated a cohort of 100 subjects who participated in the National Lung Screening Trial and had established pathological diagnoses. Data from 30 subjects were randomly selected for training and the remaining was used for testing. Participants were evaluated based on the area under the receiver operating characteristic curve (AUC) of nodule-wise malignancy scores generated by their algorithms on the test set. The challenge had 17 participants, with 11 teams submitting reports with method description, mandated by the challenge rules. Participants used quantitative methods, resulting in a reporting test AUC ranging from 0.698 to 0.913. The top five contestants used deep learning approaches, reporting an AUC between 0.87 - 0.91. The team's predictor did not achieve significant differences from each other nor from a volume change estimate (p =.05 with Bonferroni-Holm's correction).

The advent of increasingly sophisticated medical technology, surgical interventions, and supportive healthcare measures is raising survival probabilities for babies born premature and/or with life-threatening health conditions. In the United States, this trend is associated with greater numbers of neonatal surgeries and higher admission rates into neonatal intensive care units (NICU) for newborns at all birth weights. Following surgery, current pain management in NICU relies primarily on narcotics (opioids) such as morphine and fentanyl (about 100 times more potent than morphine) that lead to a number of complications, including prolonged stays in NICU for opioid withdrawal. In this paper, we review current practices and challenges for pain assessment and treatment in NICU and outline ongoing efforts using Artificial Intelligence (AI) to support pain- and opioid-sparing approaches for newborns in the future. A major focus for these next-generation approaches to NICU-based pain management is proactive pain mitigation (avoidance) aimed at preventing harm to neonates from both postsurgical pain and opioid withdrawal. AI-based frameworks can use single or multiple combinations of continuous objective variables, that is, facial and body movements, crying frequencies, and physiological data (vital signs), to make high-confidence predictions about time-to-pain onset following postsurgical sedation. Such predictions would create a therapeutic window prior to pain onset for mitigation with non-narcotic pharmaceutical and nonpharmaceutical interventions. These emerging AI-based strategies have the potential to minimize or avoid damage to the neonate's body and psyche from postsurgical pain and opioid withdrawal.