The scaling of neural networks with increasing data and model sizes necessitates the development of more efficient deep learning algorithms. A significant challenge in neural network training is the memory footprint associated with activation tensors, particularly in pointwise nonlinearity layers that traditionally save the entire input tensor for the backward pass, leading to substantial memory consumption. In this paper, we propose a modification to the handling of activation tensors in pointwise nonlinearity layers. Our method involves saving the output tensor instead of the input tensor during the forward pass. Since the subsequent layer typically also saves its input tensor, this approach reduces the total memory required by storing only one tensor between layers instead of two. This optimization is especially beneficial for transformer-based architectures like GPT, BERT, Mistral, and Llama. To enable this approach, we utilize the inverse function of the nonlinearity during the backward pass. As the inverse cannot be computed analytically for most nonlinearities, we construct accurate approximations using simpler functions. Experimental results demonstrate that our method significantly reduces memory usage without affecting training accuracy or computational performance. Our implementation is provided as a drop-in replacement for standard nonlinearity layers in the PyTorch framework, facilitating easy adoption without requiring architectural modifications.

Nearest-neighbor search in large vector databases is crucial for various machine learning applications. This paper introduces a novel method using **tensor-train** (TT) low-rank tensor decomposition to efficiently represent point clouds and enable fast approximate nearest-neighbor searches. We propose a probabilistic interpretation and utilize density estimation losses like Sliced Wasserstein to train TT decompositions, resulting in robust point cloud compression. We reveals an inherent hierarchical structure within TT point clouds, facilitating efficient approximate nearest-neighbor searches. In our paper, we provide detailed insights into the methodology and conduct comprehensive comparisons with existing methods. We demonstrate its effectiveness in various scenarios, including out-of-distribution (OOD) problems and approximate nearest-neighbor (ANN) search tasks.

Large-scale transformer models have shown remarkable performance in language modelling tasks. However, such models feature billions of parameters, leading to difficulties in their deployment and prohibitive training costs from scratch. To reduce the number of the parameters in the GPT-2 architecture, we replace the matrices of fully-connected layers with the corresponding Tensor Train Matrix~(TTM) structure. Finally, we customize forward and backward operations through the TTM-based layer for simplicity and the stableness of further training. % The resulting GPT-2-based model stores up to 40% fewer parameters, showing the perplexity comparable to the original model. On the downstream tasks, including language understanding and text summarization, the model performs similarly to the original GPT-2 model. The proposed tensorized layers could be used to efficiently pre-training other Transformer models.

The tensor train (TT) format, widely used in computational mathematics and machine learning, offers a computationally efficient method for handling multidimensional arrays, vectors, matrices, and discretized functions in various applications. In this article, we propose a new algorithm for estimating minimum/maximum elements of TT-tensors, which leads to accurate approximations. The method consists of sequential tensor multiplications of the TT-cores with an intelligent selection of candidates for the optimum. We propose a probabilistic interpretation of the method and estimate its complexity and convergence. We perform extensive numerical experiments with random tensors and various multivariable benchmark functions with the number of input dimensions up to 100. Our approach generates a solution close to the exact optimum for all model problems on a regular laptop.

Memory footprint is one of the main limiting factors for large neural network training. In backpropagation, one needs to store the input to each operation in the computational graph. Every modern neural network model has quite a few pointwise nonlinearities in its architecture, and such operation induces additional memory costs which -- as we show -- can be significantly reduced by quantization of the gradients. We propose a systematic approach to compute optimal quantization of the retained gradients of the pointwise nonlinear functions with only a few bits per each element. We show that such approximation can be achieved by computing optimal piecewise-constant approximation of the derivative of the activation function, which can be done by dynamic programming. The drop-in replacements are implemented for all popular nonlinearities and can be used in any existing pipeline. We confirm the memory reduction and the same convergence on several open benchmarks.

Tensor train (TT) format is a common approach for computationally efficient work with multidimensional arrays, vectors, matrices, and discretized functions in a wide range of applications, including computational mathematics and machine learning. In this work, we propose a new algorithm for TT-tensor optimization, which leads to very accurate approximations for the minimum and maximum tensor element. The method consists in sequential tensor multiplications of the TT-cores with an intelligent selection of candidates for the optimum. We propose the probabilistic interpretation of the method, and make estimates on its complexity and convergence. We perform extensive numerical experiments with random tensors and various multivariable benchmark functions with the number of input dimensions up to $100$. Our approach generates a solution close to the exact optimum for all model problems, while the running time is no more than $50$ seconds on a regular laptop.

The amount of generated, collected and labelled data rapidly increases nowadays, which raises the question of creating methods for extracting dataset subsets, learning on which a model can achieve the same generalization level as when learning on the whole available set of data. Methods of the optimal design family solve this problem, but they are poorly applicable to modern multidimensional data and complex models. In this paper, we construct a method based on an intuitive way to measure the quality of the dataset subset, propose ways to approximate this functional and develop an efficient optimization method to reduce the dataset size. We demonstrate the capabilities of our approach on the MNIST dataset in application to the logistic regression.

Estimation of probability density function from samples is one of the central problems in statistics and machine learning. Modern neural network-based models can learn high dimensional distributions but have problems with hyperparameter selection and are often prone to instabilities during training and inference. We propose a new efficient tensor train-based model for density estimation (TTDE). Such density parametrization allows exact sampling, calculation of cumulative and marginal density functions, and partition function. It also has very intuitive hyperparameters. We develop an efficient non-adversarial training procedure for TTDE based on the Riemannian optimization. Experimental results demonstrate the competitive performance of the proposed method in density estimation and sampling tasks, while TTDE significantly outperforms competitors in training speed.

We have analyzed codon usage for poly-Q stretches of different lengths for the human proteome. First, we have obtained that all long poly-Q stretches in Protein Data Bank (PDB) belong to the disordered regions. Second, we have found the bias for codon usage for glutamine homo-repeats in the human proteome. In the cases when the same codon is used for poly-Q stretches only CAG triplets are found. Similar results are obtained for human proteins with glutamine homo-repeats associated with diseases. Moreover, for proteins associated with diseases (from the HraDis database), the fraction of proteins for which the same codon is used for glutamine homo-repeats is less (22%) than for proteins from the human proteome (26%). We have demonstrated for poly-Q stretches in the human proteome that in some cases (28) the splicing sites correspond to the homo-repeats and in 11 cases, these sites appear at the C -terminal part of the homo-repeats with statistical significance 10 -8 .