Mamba state-space models (SSMs) have recently outperformed state-of-the-art (SOTA) Transformer large language models (LLMs) in various tasks and been widely adapted. However, a major concern for stable learning in recurrent-based deep models (such as SSMs) is the sensitivity of their recurrent dynamics. Despite widespread adaptation, the sensitivity of Mamba's recurrent dynamics under common fine-tuning methods-e.g., mixed-precision fine-tuning (MPFT) and parameter-efficient fine-tuning (PEFT)-remains unexplored. Empirically, we show that Mamba LLMs are extremely stable to changes introduced by combinations of MPFT and PEFT, in stark contrast to Transformer LLMs, which we demonstrate may drastically diverge from their respective full-precision counterparts under different combinations of MPFT and PEFT (despite the near-ubiquitous adaptation of these fine-tuning frameworks for attention-based models). The demonstrated robustness of Mamba LLMs are due to their recurrent dynamics, which we prove are guaranteed to be stable using dynamical systems theory (in particular, Lyapunov stability). We conclude by using MPFT and PEFT to novelly study Mamba LLMs' in-context learning (ICL) abilities on natural language tasks, thus supplementing other recent work.

Mamba state-space models (SSMs) have recently outperformed state-of-the-art (SOTA) Transformer large language models (LLMs) in various tasks and been widely adapted. However, Mamba’s downstream learning capabilities remain either unexplored–e.g., mixed-precision (MPFT) and parameter-efficient fine-tuning (PEFT)–or under-evaluated–e.g., in-context learning (ICL). For the latter, recent works reported Mamba’s ICL rivals SOTA Transformer LLMs using non-standard benchmarks. In contrast, we show that on standard benchmarks, pretrained Mamba models achieve only 38% of the ICL performance improvements (over zero-shot) of comparable Transformers. Enabling MPFT and PEFT in Mamba architectures is challenging due to recurrent dynamics and highly customized CUDA kernels, respectively. However, we prove that Mamba’s recurrent dynamics are robust to small input changes using dynamical systems theory. Empirically, we show that performance changes in Mamba’s inference and fine-tuning due to mixed-precision align with Transformer LLMs. Furthermore, we prove that targeting key memory buffers in Mamba’s customized CUDA kernels for low-rank adaptation regularizes SSM parameters, thus achieving parameter efficiency while retaining speedups. We show that combining MPFT and PEFT enables up to 2.15 times more tokens-per-second and 65.5% reduced memory-per-token compared to full Mamba fine-tuning, while achieving up to 81.5% of the ICL performance improvements (over zero-shot) of comparably fine-tuned Transformers.

One of the most efficient methods to solve L2-regularized primal problems, such as logistic regression and linear support vector machine (SVM) classification, is the widely used trust region Newton algorithm, TRON. While TRON has recently been shown to enjoy substantial speedups on shared-memory multi-core systems, exploiting graphical processing units (GPUs) to speed up the method is significantly more difficult, owing to the highly complex and heavily sequential nature of the algorithm. In this work, we show that using judicious GPU-optimization principles, TRON training time for different losses and feature representations may be drastically reduced. For sparse feature sets, we show that using GPUs to train logistic regression classifiers in LIBLINEAR is up to an order-of-magnitude faster than solely using multithreading. For dense feature sets--which impose far more stringent memory constraints--we show that GPUs substantially reduce the lengthy SVM learning times required for state-of-the-art proteomics analysis, leading to dramatic improvements over recently proposed speedups. Furthermore, we show how GPU speedups may be mixed with multithreading to enable such speedups when the dataset is too large for GPU memory requirements; on a massive dense proteomics dataset of nearly a quarter-billion data instances, these mixed-architecture speedups reduce SVM analysis time from over half a week to less than a single day while using limited GPU memory.

Semi-supervised machine learning post-processors critically improve peptide identification of shot-gun proteomics data. Such post-processors accept the peptide-spectrum matches (PSMs) and feature vectors resulting from a database search, train a machine learning classifier, and recalibrate PSMs using the trained parameters, often yielding significantly more identified peptides across q -value thresholds. However, current state-of-the-art post-processors rely on shallow machine learning methods, such as support vector machines. In contrast, the powerful training capabilities of deep learning models have displayed superior performance to shallow models in an ever-growing number of other fields. In this work, we show that deep models significantly improve the recalibration of PSMs compared to the most accurate and widely-used post-processors, such as Percolator and PeptideProphet. Furthermore, we show that deep learning is able to adaptively analyze complex datasets and features for more accurate universal post-processing, leading to both improved Prosit analysis and markedly better recalibration of recently developed database-search functions.### Competing Interest StatementThe authors have declared no competing interest.

The processing of peptide tandem mass spectrometry data involves matching observed spectra against a sequence database. The ranking and calibration of these peptide-spectrum matches can be improved substantially using a machine learning postprocessor. Here, we describe our efforts to speed up one widely used postprocessor, Percolator. The improved software is dramatically faster than the previous version of Percolator, even when using relatively few processors. We tested the new version of Percolator on a data set containing over 215 million spectra and recorded an overall reduction to 23% of the running time as compared to the unoptimized code. We also show that the memory footprint required by these speedups is modest relative to that of the original version of Percolator.

Percolator is an important tool for greatly improving the results of a database search and subsequent downstream analysis. Using support vector machines (SVMs), Percolator recalibrates peptide-spectrum matches based on the learned decision boundary between targets and decoys. To improve analysis time for large-scale data sets, we update Percolator's SVM learning engine through software and algorithmic optimizations rather than heuristic approaches that necessitate the careful study of their impact on learned parameters across different search settings and data sets. We show that by optimizing Percolator's original learning algorithm, l2-SVM-MFN, large-scale SVM learning requires nearly only a third of the original runtime. Furthermore, we show that by employing the widely used Trust Region Newton (TRON) algorithm instead of l2-SVM-MFN, large-scale Percolator SVM learning is reduced to nearly only a fifth of the original runtime. Importantly, these speedups only affect the speed at which Percolator converges to a global solution and do not alter recalibration performance. The upgraded versions of both l2-SVM-MFN and TRON are optimized within the Percolator codebase for multithreaded and single-thread use and are available under Apache license at bitbucket.org/jthalloran/percolator_upgrade .

This paper introduces Jensen, an easily extensible and scalable toolkit for production-level machine learning and convex optimization. Jensen implements a framework of convex (or loss) functions, convex optimization algorithms (including Gradient Descent, L-BFGS, Stochastic Gradient Descent, Conjugate Gradient, etc.), and a family of machine learning classifiers and regressors (Logistic Regression, SVMs, Least Square Regression, etc.). This framework makes it possible to deploy and train models with a few lines of code, and also extend and build upon this by integrating new loss functions and optimization algorithms.

Tandem mass spectrometry (MS/MS) is a high-throughput technology used to identify the proteins present in a complex, biological sample. Critical to MS/MS is the ability to accurately identify the peptide responsible for producing each observed spectrum. Recently, a dynamic Bayesian network (DBN) approach was shown to achieve state-of-the-art accuracy for this peptide identification problem. Modeling the stochastic process by which a peptide produces an MS/MS spectrum, this DBN for Rapid Identification of Peptides (DRIP) uses probabilistic inference to efficiently determine the most probable alignment between a peptide and an observed spectrum. DRIP's dynamic alignment strategy improves upon standard "static" alignment strategies, which rely on fixed quantization of the temporal axis of MS/MS data, in several significant ways. In particular, DRIP allows learning non-linear shifts of the temporal axis and, owing to the generative nature of the model, accurate feature extraction for substantially improved discriminative analysis (i.e., Percolator post-processing), all of which are supported in the DRIP Toolkit (DTK). Herein we describe how DTK may be used to significantly improve MS/MS identification accuracy, as well as DTK's interactive features for fine-grained analysis, including on the fly inference and plotting attributes.

The most widely used technology to identify the proteins present in a complex biological sample is tandem mass spectrometry, which quickly produces a large collection of spectra representative of the peptides (i.e., protein subsequences) present in the original sample. In this work, we greatly expand the parameter learning capabilities of a dynamic Bayesian network (DBN) peptide-scoring algorithm, Didea [25], by deriving emission distributions for which its conditional log-likelihood scoring function remains concave. We show that this class of emission distributions, called Convex Virtual Emissions (CVEs), naturally generalizes the log-sum-exp function while rendering both maximum likelihood estimation and conditional maximum likelihood estimation concave for a wide range of Bayesian networks. Utilizing CVEs in Didea allows efficient learning of a large number of parameters while ensuring global convergence, in stark contrast to Didea's previous parameter learning framework (which could only learn a single parameter using a costly grid search) and other trainable models [12, 13, 14] (which only ensure convergence to local optima). The newly trained scoring function substantially outperforms the state-of-the-art in both scoring function accuracy and downstream Fisher kernel analysis. Furthermore, we significantly improve Didea's runtime performance through successive optimizations to its message passing schedule and derive explicit connections between Didea's new concave score and related MS/MS scoring functions.

In the past two decades, the field of proteomics has seen explosive growth, largely due to the development of tandem mass spectrometry (MS/MS). With a complex biological sample as input, a typical MS/MS experiment quickly produces a large (often numbering in the hundreds-of-thousands) collection of spectra representative of the proteins present in the original complex sample. A majority of widely used methods to search and identify MS/MS spectra use scoring functions which rely on static, hand-selected parameters rather than affording the ability to learn parameters and adapt to the widely varying characteristics of MS/MS data. In this talk, we discuss recent work utilizing dynamic Bayesian networks (DBNs) to identify MS/MS spectra. In particular, we discuss a recently proposed DBN for Rapid Identification of Peptides (DRIP) which, in contrast to popular scoring functions, allows efficient generative and discriminative learning of parameters to achieve state-of-theart spectrum-identification accuracy. Furthermore, facilitated by DRIP’s generative nature, we present current innovations leveraging DBNs to significantly enhance many other aspects of MS/MS analysis, such as improving downstream discriminative classification via detailed feature extraction and speeding up identification runtime using trellises and approximate inference.

A comprehensive characterization of tumor genetic heterogeneity is critical for understanding how cancers evolve and escape treatment. Although many algorithms have been developed for capturing tumor heterogeneity, they are designed for analyzing either a single type of genomic aberration or individual biopsies. Here we present THEMIS (Tumor Heterogeneity Extensible Modeling via an Integrative System), which allows for the joint analysis of different types of genomic aberrations from multiple biopsies taken from the same patient, using a dynamic graphical model. Simulation experiments demonstrate higher accuracy of THEMIS over its ancestor, TITAN. The heterogeneity analysis results from THEMIS are validated with single cell DNA sequencing from a clinical tumor biopsy. When THEMIS is used to analyze tumor heterogeneity among multiple biopsies from the same patient, it helps to reveal the mutation accumulation history, track cancer progression, and identify the mutations related to treatment resistance. We implement our model via an extensible modeling platform, which makes our approach open, reproducible, and easy for others to extend.

In the past two decades, the field of proteomics has seen explosive growth, largely due to the development of tandem mass spectrometry (MS/MS). With a complex biological sample as input, a typical MS/MS experiment quickly produces a large (often numbering in the hundreds-of-thousands) collection of spectra representative of the proteins present in the original complex sample. A majority of widely used methods to search and identify MS/MS spectra use scoring functions which rely on static, hand-selected parameters rather than affording the ability to learn parameters and adapt to the widely varying characteristics of MS/MS data. In this talk, we discuss recent work utilizing dynamic Bayesian networks (DBNs) to identify MS/MS spectra. In particular, we discuss a recently proposed DBN for Rapid Identification of Peptides (DRIP) which, in contrast to popular scoring functions, allows efficient generative and discriminative learning of parameters to achieve state-of-theart spectrum-identification accuracy. Furthermore, facilitated by DRIP’s generative nature, we present current innovations leveraging DBNs to significantly enhance many other aspects of MS/MS analysis, such as improving downstream discriminative classification via detailed feature extraction and speeding up identification runtime using trellises and approximate inference.

Tandem mass spectrometry (MS/MS) is a high-throughput technology used to identify the proteins in a complex biological sample, such as a drop of blood. A collection of spectra is generated at the output of the process, each spectrum of which is representative of a peptide (protein subsequence) present in the original complex sample. In this work, we leverage the log-likelihood gradients of generative models to improve the identification of such spectra. In particular, we show that the gradient of a recently proposed dynamic Bayesian network (DBN) [7] may be naturally employed by a kernel-based discriminative classifier. The resulting Fisher kernel substantially improves upon recent attempts to combine generative and discriminative models for post-processing analysis, outperforming all other methods on the evaluated datasets. We extend the improved accuracy offered by the Fisher kernel framework to other search algorithms by introducing Theseus, a DBN representing a large number of widely used MS/MS scoring functions. Furthermore, with gradient ascent and max-product inference at hand, we use Theseus to learn model parameters without any supervision.

Graphical models (GMs) provide a flexible framework for modeling phenomena. In the past few decades, GMs have become indispensable tools for machine learning and computational biology. They afford a wide range of modeling granularity, from restricting only exact events of interest to occur to allowing all possible events in a phenomenon’s event space. For all such modeling considerations, GMs afford efficient algorithms to perform inference over the probabilistic quantities of interest. In this thesis, we show how GMs may be leveraged to improve both identification accuracy and search runtime of tandem mass (MS/MS) spectra. For the majority of existing MS/MS scoring algorithms, we give equivalent GMs and show how search time may be algorithmically improved. We present GMs for posterior based (sum-product) and max-product inference which offer state-of-the-art performance and, most importantly, are amenable to efficient parameter estimation. Furthermore, we show how a GM which generatively models the stochastic process by which peptides produce MS/MS spectra may be utilized to calculate features for improved classification between correctly and incorrectly identified spectra, leading to significantly improved identification accuracy.

A central problem in mass spectrometry analysis involves identifying, for each observed tandem mass spectrum, the corresponding generating peptide. We present a dynamic Bayesian network (DBN) toolkit that addresses this problem by using a machine learning approach. At the heart of this toolkit is a DBN for Rapid Identification (DRIP), which can be trained from collections of high-confidence peptide-spectrum matches (PSMs). DRIP's score function considers fragment ion matches using Gaussians rather than fixed fragment-ion tolerances and also finds the optimal alignment between the theoretical and observed spectrum by considering all possible alignments, up to a threshold that is controlled using a beam-pruning algorithm. This function not only yields state-of-the art database search accuracy but also can be used to generate features that significantly boost the performance of the Percolator postprocessor. The DRIP software is built upon a general purpose DBN toolkit (GMTK), thereby allowing a wide variety of options for user-specific inference tasks as well as facilitating easy modifications to the DRIP model in future work. DRIP is implemented in Python and C++ and is available under Apache license at http://melodi-lab.github.io/dripToolkit .

Tandem mass spectrometry (MS/MS) is the dominant high throughput technology for identifying and quantifying proteins in complex biological samples. Analysis of the tens of thousands of fragmentation spectra produced by an MS/MS experiment begins by assigning to each observed spectrum the peptide that is hypothesized to be responsible for generating the spectrum. This assignment is typically done by searching each spectrum against a database of peptides. To our knowledge, all existing MS/MS search engines compute scores individually between a given observed spectrum and each possible candidate peptide from the database. In this work, we use a trellis, a data structure capable of jointly representing a large set of candidate peptides, to avoid redundantly recomputing common sub-computations among different candidates. We show how trellises may be used to significantly speed up existing scoring algorithms, and we theoretically quantify the expected speedup afforded by trellises. Furthermore, we demonstrate that compact trellis representations of whole sets of peptides enables efficient discriminative learning of a dynamic Bayesian network for spectrum identification, leading to greatly improved spectrum identification accuracy.

The Graphical Models ToolKit (GMTK) is a powerful and flexible prototyping language for designing dynamic Bayesian networks (DBNs). This tutorial is meant to help new users’ understanding of GMTK by presenting hidden Markov models (HMMs) which make use of some of the software’s large number of features. Generative and discriminative training approaches supported in GMTK are discussed with relevant examples, as well as testing using an HMM for a simple classification task. All described models and scripts are available in the tarball housing this document. The following examples are intended for those interested in using the Graphical Models ToolKit (GMTK) to perform various forms of inference/learning utilizing graphical models. Using GMTK, we will first train and test a hidden Markov model (HMM) whose emission states are discrete (Section 1), then train and test one whose emission states are real valued (conditionally Gaussian) (Section 2). Also discussed are generative and discriminative training approaches supported in GMTK. 1 Classifying the weather: discrete observations Consider the scenario wherein the area you work/reside only encounters three types of weather: sunny, rainy, and foggy. Assume that you go to work in an office everyday, and during your working hours you do not get to see nor experience a given days weather (you’re office also has no windows looking outside; this was my office in Hawaii ironically enough). Curiosity mounts as time goes on and you become more and more curious about what the daily weather is. Your hope lies in that you have an office mate who comes in hours later than you, and this office mate brings in an umbrella some days, and no umbrella all other days. In the framework of the HMM, the hidden layer is the state of the weather (indeed, it is hidden from you). The observed layer consists of the daily observations of weather your office mate has brought

We present a peptide-spectrum alignment strategy that employs a dynamic Bayesian network (DBN) for the identification of spectra produced by tandem mass spectrometry (MS/MS). Our method is fundamentally generative in that it models peptide fragmentation in MS/MS as a physical process. The model traverses an observed MS/MS spectrum and a peptide-based theoretical spectrum to calculate the best alignment between the two spectra. Unlike all existing state-of-the-art methods for spectrum identification that we are aware of, our method can learn alignment probabilities given a dataset of high-quality peptide-spectrum pairs. The method, moreover, accounts for noise peaks and absent theoretical peaks in the observed spectrum. We demonstrate that our method outperforms, on a majority of datasets, several widely used, state-of-the-art database search tools for spectrum identification. Furthermore, the proposed approach provides an extensible framework for MS/MS analysis and provides useful information that is not produced by other methods, thanks to its generative structure.

Liquid chromatography coupled with tandem mass spectrometry, also known as shotgun proteomics, is a widely-used high-throughput technology for identifying proteins in complex biological samples. Analysis of the tens of thousands of fragmentation spectra produced by a typical shotgun proteomics experiment begins by assigning to each observed spectrum the peptide hypothesized to be responsible for generating the spectrum, typically done by searching each spectrum against a database of peptides. We have recently described a machine learning method---Dynamic Bayesian Network for Rapid Identification of Peptides (DRIP)---that not only achieves state-of-the-art spectrum identification performance on a variety of datasets but also provides a trainable model capable of returning valuable auxiliary information regarding specific peptide-spectrum matches. In this work, we present two significant improvements to DRIP. First, we describe how to use word lattices, which are widely used in natural language processing, to significantly speed up DRIP's computations. To our knowledge, all existing shotgun proteomics search engines compute independent scores between a given observed spectrum and each possible candidate peptide from the database. The key idea of the word lattice is to represent the set of candidate peptides in a single data structure, thereby allowing sharing of redundant computations among the different candidates. We demonstrate that using lattices in conjunction with DRIP leads to speedups on the order of tens across yeast and worm data sets. Second, we introduce a variant of DRIP that uses a discriminative training framework, performing maximum mutual entropy estimation rather than maximum likelihood estimation. This modification improves DRIP's statistical power, enabling us to increase the number of identified spectrum at a 1% false discovery rate on yeast and worm data sets.

We present a peptide-spectrum alignment strategy that employs a dynamic Bayesian network (DBN) for the identification of spectra produced by tandem mass spectrometry (MS/MS). Our method is fundamentally generative in that it models peptide fragmentation in MS/MS as a physical process. The model traverses an observed MS/MS spectrum and a peptide-based theoretical spectrum to calculate the best alignment between the two spectra. Unlike all existing state-of-the-art methods for spectrum identification that we are aware of, our method can learn alignment probabilities given a dataset of high-quality peptide-spectrum pairs. The method, moreover, accounts for noise peaks and absent theoretical peaks in the observed spectrum. We demonstrate that our method out-performs, on a majority of datasets, several widely used, state-of-the-art database search tools for spectrum identification. Furthermore, the proposed approach provides an extensible framework for MS/MS analysis and provides useful information that is not produced by other methods, thanks to its generative structure.