Recent generative data augmentation methods conditioned on both image and text prompts struggle to balance between fidelity and diversity, as it is challenging to preserve essential image details while aligning with varied text prompts. This challenge arises because representations in the synthesis process often become entangled with non-essential input image attributes such as environmental contexts, creating conflicts with text prompts intended to modify these elements. To address this, we propose a personalized image generation framework that uses a salient concept-aware image embedding model to reduce the influence of irrelevant visual details during the synthesis process, thereby maintaining intuitive alignment between image and text inputs. By generating images that better preserve class-discriminative features with additional controlled variations, our framework effectively enhances the diversity of training datasets and thereby improves the robustness of downstream models. Our approach demonstrates superior performance across eight fine-grained vision datasets, outperforming state-of-the-art augmentation methods with averaged classification accuracy improvements by 0.73\% and 6.5\% under conventional and long-tail settings, respectively.

Existing losses used in deep metric learning (DML) for image retrieval often lead to highly non-uniform intra-class and inter-class representation structures across test classes and data distributions. When combined with the common practice of using a fixed threshold to declare a match, this gives rise to significant performance variations in terms of false accept rate (FAR) and false reject rate (FRR) across test classes and data distributions. We define this issue in DML as threshold inconsistency. In real-world applications, such inconsistency often complicates the threshold selection process when deploying commercial image retrieval systems. To measure this inconsistency, we propose a novel variance-based metric called Operating-Point-Inconsistency-Score (OPIS) that quantifies the variance in the operating characteristics across classes. Using the OPIS metric, we find that achieving high accuracy levels in a DML model does not automatically guarantee threshold consistency. In fact, our investigation reveals a Pareto frontier in the high-accuracy regime, where existing methods to improve accuracy often lead to degradation in threshold consistency. To address this trade-off, we introduce the Threshold-Consistent Margin (TCM) loss, a simple yet effective regularization technique that promotes uniformity in representation structures across classes by selectively penalizing hard sample pairs. Extensive experiments demonstrate TCM's effectiveness in enhancing threshold consistency while preserving accuracy, simplifying the threshold selection process in practical DML settings.

We present a sequence-to-sequence vision-language model whose parameters are jointly trained on all tasks and fully shared among multiple tasks, resulting in a single model which we named Musketeer. The integration of knowledge across heterogeneous tasks is enabled by a novel feature called Task Explanation Prompt (TEP). TEP reduces interference among tasks, allowing the model to focus on their shared structure. With a single model, Musketeer achieves results comparable to or better than strong baselines trained on single tasks, almost uniformly across multiple tasks.

The open world is inherently dynamic, characterized by ever-evolving concepts and distributions. Continual learning (CL) in this dynamic open-world environment presents a significant challenge in effectively generalizing to unseen test-time classes. To address this challenge, we introduce a new practical CL setting tailored for open-world visual representation learning. In this setting, subsequent data streams systematically introduce novel classes that are disjoint from those seen in previous training phases, while also remaining distinct from the unseen test classes. In response, we present Dynamic Prompt and Representation Learner (DPaRL), a simple yet effective Prompt-based CL (PCL) method. Our DPaRL learns to generate dynamic prompts for inference, as opposed to relying on a static prompt pool in previous PCL methods. In addition, DPaRL jointly learns dynamic prompt generation and discriminative representation at each training stage whereas prior PCL methods only refine the prompt learning throughout the process. Our experimental results demonstrate the superiority of our approach, surpassing state-of-the-art methods on well-established open-world image retrieval benchmarks by an average of 4.7% improvement in Recall@1 performance.

We present a vision-language model whose parameters are jointly trained on all tasks and fully shared among multiple heterogeneous tasks which may interfere with each other, resulting in a single model which we named Musketeer. The integration of knowledge across heterogeneous tasks is enabled by a novel feature called Task Explanation Prompt (TEP). With rich and structured information such as task input/output format, TEP reduces interference among tasks, allowing the model to focus on their shared structure. With a single model, Musketeer achieves results comparable to or better than strong baselines trained on single tasks, almost uniformly across multiple tasks.

Existing losses used in deep metric learning (DML) for image retrieval often lead to highly non-uniform intra-class and inter-class representation structures across test classes and data distributions. When combined with the common practice of using a fixed threshold to declare a match, this gives rise to significant performance variations in terms of false accept rate (FAR) and false reject rate (FRR) across test classes and data distributions. We define this issue in DML as threshold inconsistency. In real-world applications, such inconsistency often complicates the threshold selection process when deploying commercial image retrieval systems. To measure this inconsistency, we propose a novel variance-based metric called Operating-Point-Inconsistency-Score (OPIS) that quantifies the variance in the operating characteristics across classes. Using the OPIS metric, we find that achieving high accuracy levels in a DML model does not automatically guarantee threshold consistency. In fact, our investigation reveals a Pareto frontier in the high-accuracy regime, where existing methods to improve accuracy often lead to degradation in threshold consistency. To address this trade-off, we introduce the Threshold-Consistent Margin (TCM) loss, a simple yet effective regularization technique that promotes uniformity in representation structures across classes by selectively penalizing hard sample pairs. Extensive experiments demonstrate TCM's effectiveness in enhancing threshold consistency while preserving accuracy, simplifying the threshold selection process in practical DML settings.

We present a vision-language model whose parameters are jointly trained on all tasks and fully shared among multiple heterogeneous tasks which may interfere with each other, resulting in a single model which we named Musketeer. The integration of knowledge across heterogeneous tasks is enabled by a novel feature called Task Explanation Prompt (TEP). With rich and structured information such as task input/output format, TEP reduces interference among tasks, allowing the model to focus on their shared structure. With a single model, Musketeer achieves results comparable to or better than strong baselines trained on single tasks, almost uniformly across multiple tasks.

Convolutional neural networks are widely used in practical application in many diverse environments. Each different environment requires a different optimized network to maximize accuracy under its unique hardware constraints and latency requirements. To find models for this varied array of potential deployment targets, once-for-all (OFA) was introduced as a way to simultaneously co-train many models at once, while keeping the total training cost constant. However, the total training cost is very high, requiring up to 1200 GPU-hours. Compound OFA (compOFA) decreased the training cost of OFA by 2$\times$ by coupling model dimensions to reduce the search space of possible models by orders of magnitude, while also simplifying the training procedure. In this work, we continue the effort to reduce the training cost of OFA methods. While both OFA and compOFA use a pre-trained teacher network, we propose an in-place knowledge distillation procedure to train the super-network simultaneously with the sub-networks. Within this in-place distillation framework, we develop an upper-attentive sample technique that reduces the training cost per epoch while maintaining accuracy. Through experiments on ImageNet, we demonstrate that, we can achieve a $2\times$ - $3\times$ ($1.5\times$ - $1.8\times$) reduction in training time compared to the state of the art OFA and compOFA, respectively, without loss of optimality.

With the shift towards on-device deep learning, ensuring a consistent behavior of an AI service across diverse compute platforms becomes tremendously important. Our work tackles the emergent problem of reducing predictive inconsistencies arising as negative flips: test samples that are correctly predicted by a less accurate model, but incorrectly by a more accurate one. We introduce REGression constrained Neural Architecture Search (REG-NAS) to design a family of highly accurate models that engender fewer negative flips. REG-NAS consists of two components: (1) A novel architecture constraint that enables a larger model to contain all the weights of the smaller one thus maximizing weight sharing. This idea stems from our observation that larger weight sharing among networks leads to similar sample-wise predictions and results in fewer negative flips; (2) A novel search reward that incorporates both Top-1 accuracy and negative flips in the architecture search metric. We demonstrate that \regnas can successfully find desirable architectures with few negative flips in three popular architecture search spaces. Compared to the existing state-of-the-art approach, REG-NAS enables 33-48% relative reduction of negative flips.

The choice of data type has a major impact on speed, accuracy, and power consumption of deep learning accelerators. Quantizing the weights and activations of neural networks to integer based computation is an industry standard for reducing memory footprint and computation cost of inference in embedded systems. Uniform weight quantization can be used for tasks where accuracy drop can be tolerated. However, the drop in accuracy due to a uniform quantization might be non-negligible especially when performed on shallow networks, complex computer vision tasks, or with lower-bit integers. In this paper, we introduce a software and a hardware solution to improve on a baseline integer based uniform quantization so that it can be run on lower power systems and with even less bits. We also introduce a novel encoding technique on top of our software solution specific for partial sums to significantly reduce memory footprint, latency and energy consumption due to movement of partial sums. The proposed SW solution exploits non-uniform piece-wise linear quantization to improve accuracy by capturing bell shaped distribution of weights while still using INT-based computation units. The proposed partial sum encoding can be applied to the partial sums regardless of uniform or non-uniform quantization. The proposed HW solution can either combine integers to make larger integers or turn them into Floating-Point operations so that various levels can have various precisions or data types, if necessary. To do so, we studied upper limits of precision we need in our compute units to support floating point inner product operations. It turns out that we can improve upon integer IPUs to perform accurate floating-point operations without introducing large shift units or wide adder trees. Our proposed SW solution (PWLQ) achieves the state-of-the-art results on all cases and it outperforms all other methods with a large margin. The proposed partial sum encoding technique effectively compresses the partial sum of networks like Resnet-50 down to 12bits (from Int64/32) without loss in accuracy. The proposed HW architecture achieves area improvements of up to 46% in TOPS/mm2 with power efficiency improvements of up to 63% in TOPS/W when compared to state of the art mixed precision implementation.

In this paper, we provide an in-depth study of Stochastic Backpropagation (SBP) when training deep neural networks for standard image classification and object detection tasks. During backward propagation, SBP calculates gradients by using only a subset of feature maps to save GPU memory and computational cost. We interpret SBP as an efficient way to implement stochastic gradient decent by performing backpropagation dropout, which leads to significant memory saving and training run-time reduction, with a minimal impact on the overall model accuracy. We offer best practices to apply SBP for training image recognition models, which can be adopted in learning a wide range of deep neural networks. Experiments on image classification and object detection show that SBP can save up to 40% of GPU memory with less than 1% accuracy degradation. Code is available at: https://github.com/amazon-research/stochastic-backpropagation

Quantization plays an important role in the energy-efficient deployment of deep neural networks on resource-limited devices. Post-training quantization is highly desirable since it does not require retraining or access to the full training dataset. The well-established uniform scheme for post-training quantization achieves satisfactory results by converting neural networks from full-precision to 8-bit fixed-point integers. However, it suffers from significant performance degradation when quantizing to lower bit-widths. In this paper, we propose a piecewise linear quantization (PWLQ) scheme (Code will be made available at https://github.com/jun-fang/PWLQ ) to enable accurate approximation for tensor values that have bell-shaped distributions with long tails. Our approach breaks the entire quantization range into non-overlapping regions for each tensor, with each region being assigned an equal number of quantization levels. Optimal breakpoints that divide the entire range are found by minimizing the quantization error. Compared to state-of-the-art post-training quantization methods, experimental results show that our proposed method achieves superior performance on image classification, semantic segmentation, and object detection with minor overhead.

The behavior of fluid flow through rock fractures at varying temperatures is a critical issue in many subsurface geo-energy projects. Previous work has considered the thermal effects on fracture permeability, but not in isolation of the chemical effects as well. Therefore, to quantitatively assess the mechanical influences on fracture permeability, we present results from permeability tests of five cylindrical Beishan granite samples, each with a single artificial fracture, at different temperatures. Three samples were tested at a constant confining pressure of 5 MPa and temperatures of 22 degrees C, 90 degrees C and 150 degrees C for 22 days, to examine creepinduced changes in fracture permeability. Fracture permeability decreases with time until about 10 days, and the eventual magnitudes of fracture permeability reduction are much greater at 90 degrees C and 150 degrees C than at 22 degrees C. Two samples were subjected to three heating cycles (30 degrees C to 150 degrees C) at a constant confining pressure of 5 MPa. Fracture permeability de- creases as temperatures increase from 30 degrees C to 150 degrees C, and then slight changes in permeability occur as the sample is cooled to 30 degrees C. A similar tendency appears in subsequent temperature cycles, while the magnitude of fracture permeability reduction decreases with increasing heatingcooling cycles. A coupled thermalmechanical model considering asperity damage is developed to describe the thermallyinduced changes in fracture perme- ability, which properly predicts the experimental results. The residual deformation of asperities and temperature dependent Young's modulus play an important role in thermallyinduced changes in fracture permeability.

Quantization plays an important role in the energy-efficient deployment of deep neural networks on resource-limited devices. Post-training quantization is highly desirable since it does not require retraining or access to the full training dataset. The well-established uniform scheme for post-training quantization achieves satisfactory results by converting neural networks from full-precision to 8-bit fixed-point integers. However, it suffers from significant performance degradation when quantizing to lower bit-widths. In this paper, we propose a piecewise linear quantization (PWLQ) scheme to enable accurate approximation for tensor values that have bell-shaped distributions with long tails. Our approach breaks the entire quantization range into non-overlapping regions for each tensor, with each region being assigned an equal number of quantization levels. Optimal breakpoints that divide the entire range are found by minimizing the quantization error. Compared to state-of-the-art post-training quantization methods, experimental results show that our proposed method achieves superior performance on image classification, semantic segmentation, and object detection with minor overhead.

Abstract Traditional immunohistochemistry (IHC) techniques utilize one slide per biomarker. When clinical samples are precious and the number of serial sections is limited, comprehensive biomarker profiling becomes difficult with IHC. MultiOmyx is a proprietary, multiplexing methodology capable of staining up to 60 biomarkers on the same slide. The output of the assay enables quantitative profiling of tissues at a single cell level. The assay generates data for millions of cells with billions of queryable data points. To detect and classify cells efficiently at this large scale, we developed an image analysis framework using Deep Learning. Our framework consists of seven major steps: (1) manual annotation of a small subset of the nuclear staining channel (DAPI); (2) training of a fully convolutional neural network [1] on this annotation-set to generate a feature map identifying cell centers; (3) application of the trained network in (2) on the nuclear stain (DAPI) of the entire dataset to delineate individual cells; (4) manual annotation of a small subset of each of the other biomarker channels; (5) training of a convolutional neural network on these annotation-sets for binary classification of each of the biomarkers; (6) application of these classifiers to the entire dataset; (7) combination of the binary classification results to identify phenotypes of interest. For our output, we provide both visual label maps and classification summary tables for individual and co-localized biomarkers at the region of interest level and the entire slide level. In addition, combining the phenotype and location information allows us to visualize complex spatial relationships in the tissue. The benefits of using this Deep Learning framework are greatly felt through increased time efficiency without a loss in accuracy, when compared to more traditional computer vision methods requiring high levels of parameter fine-tuning. For future work, we plan on fully automating the approach as more manual annotation-sets are generated. Citation Format: Mate L. Nagy, Arezoo Hanifi, Ahalya Tirupsur, Geoffrey Wong, Jun Fang, Nicholas Hoe, Qingyan Au, Raghav K. Padmanabhan. Efficient large-scale cell classification and analysis for MultiOmyxTMassays: A deep learning approach [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2018; 2018 Apr 14-18; Chicago, IL. Philadelphia (PA): AACR; Cancer Res 2018;78(13 Suppl):Abstract nr 2256.

Tumor microenvironment (TME) consists of heterogeneous subsets of myeloid cells and plays a crucial role in promoting cancer development and metastasis. Tumor associated macrophages (TAMs) and myeloid-derived suppressor cells (MDSCs) all contribute to an immunologically permissive microenvironment for cancer cells. On basis of the expression of surface markers, MDSC can be further subdivided into granulocytic MDSC (G-MDSC, polymorphonuclear MDSC) and monocytic MDSC (M-MDSC). In solid tumors, these different myeloid cell populations are well characterized and extensively studied. However, in hematological malignancies the role of myeloid cell subsets has been less studied. A recent study showed increase in MDSC in the bone marrow (BM) at time of diagnosis in acute myeloid leukemia (AML) patients (Sun H. et al. Int J Hematol. 2015). Significantly higher numbers of G-MDSC and M-MDSC were present at diagnosis in classic Hodgkin lymphoma (cHL) (Romano A. et al. Br J Haematol. 2015). The accumulation of TAMs was also reported to be associated with poor prognosis in cHL (Steidl C. et al. N Engl J Med. 2010). Collectively, these results indicate that the tumor-resident myeloid cells play an important clinical role, thus highlighting the need for monitoring and deeper characterization of various myeloid subsets in hematological malignancies, especially in the tumor FFPE sections.

We propose a hybrid approach to solve the high-frequency Helmholtz equation with point source terms in smooth heterogeneous media. The method is based on the ray-based finite element method (ray-FEM) [29], whose original version can not handle the singularity close to point sources accurately. This pitfall is addressed by combining the ray-FEM, which is used to compute the smooth far-field of the solution accurately, with a high-order asymptotic expansion close to the point source, which is used to properly capture the singularity of the solution in the near-field. The method requires a fixed number of grid points per wavelength to accurately represent the wave field with an asymptotic convergence rate of O(ω−1/2), where ω is the frequency parameter in the Helmholtz equation. In addition, a fast sweeping-type preconditioner is used to solve the resulting linear system. We present numerical examples in 2D to show both accuracy and efficiency of our method as the frequency increases. In particular, we provide numerical evidence of the convergence rate, and we show empirically that the overall complexity is O(ω2) up to a poly-logarithmic factor.

We propose a hybrid approach to solve the high-frequency Helmholtz equation with point source terms in smooth heterogeneous media. The approach is a natural combination of an asymptotic expansion close to the point source, and an adaptively enriched finite element method based on learning dominant plane wave directions to compute the smooth far field. It requires a fixed number of grid points per wavelength to represent the wave field with an asymptotic convergence rate of $\mathcal{O}(\omega^{-1/2})$, where $\omega$ is the frequency parameter in the Helmholtz equation. In addition, a fast sweeping-type preconditioner is used to solve the resulting linear system. We present numerical examples in 2D to show both efficiency and convergence of our method as the frequency becomes larger and larger. In particular, we show empirically that the overall complexity is $\mathcal{O}(\omega^2)$ up to a poly-logarithmic factor.

We present a ray-based finite element method for the high-frequency Helmholtz equation in smooth media, whose basis is learned adaptively from the medium and source. The method requires a fixed number of grid points per wavelength to represent the wave field; moreover, it achieves an asymptotic convergence rate of \(\mathcal {O}(\omega ^{-\frac{1}{2}})\), where \(\omega \) is the frequency parameter in the Helmholtz equation. The local basis is motivated by the geometric optics ansatz and is composed of polynomials modulated by plane waves propagating in a few dominant ray directions. The ray directions are learned by processing a low-frequency wave field that probes the medium with the same source. Once the local ray directions are extracted, they are incorporated into the local basis to solve the high-frequency Helmholtz equation. This process can be continued to further improve the approximations for both local ray directions and high-frequency wave fields iteratively. Finally, a fast solver is developed for solving the resulting linear system with an empirical complexity \(\mathcal {O}(\omega ^d)\) up to a poly-logarithmic factor. Numerical examples in 2D are presented to corroborate the claims.

Author(s): Fang, Jun | Advisor(s): Zhao, Hongkai | Abstract: In this dissertation we propose a ray-based finite element method (ray-FEM) for the high-frequency Helmholtz equation in smooth media, whose basis are learned adaptively from the medium and source. The method requires a fixed number of grid points per wavelength to represent the wave field; moreover, it achieves an asymptotic convergence rate of $\mathcal{O}(\omega^{-\frac{1}{2}})$, where $\omega$ is the frequency parameter in the Helmholtz equation. The local basis are motivated by the geometric optics ansatz and are composed of polynomials modulated by plane waves propagating in a few dominant ray directions. The ray directions are learned by processing a low-frequency wave field that probes the medium with the same source. Once the local ray directions are extracted, they are incorporated into the local basis to solve the high-frequency Helmholtz equation. This process can be continued to further improve the approximations for both local ray directions and high-frequency wave fields iteratively. In addition, a fast sweeping-type preconditioner is used to solve the resulting linear system. We present numerical examples in 2D to show both efficiency and convergence of our method as the frequency becomes larger and larger. In particular, we show empirically that the overall complexity is $\mathcal{O}(\omega^2)$ up to a poly-logarithmic factor.