The efficient implementation of large language models (LLMs) is crucial for deployment on resource-constrained devices. Low-rank tensor compression techniques, such as tensor-train (TT) networks, have been widely studied for over-parameterized neural networks. However, their applications to compress pre-trained large language models (LLMs) for downstream tasks (post-training) remains challenging due to the high-rank nature of pre-trained LLMs and the lack of access to pretraining data. In this study, we investigate low-rank tensorized LLMs during fine-tuning and propose sparse augmented tensor networks (Saten) to enhance their performance. The proposed Saten framework enables full model compression. Experimental results demonstrate that Saten enhances both accuracy and compression efficiency in tensorized language models, achieving state-of-the-art performance.

The transformer architecture has revolutionized Natural Language Processing (NLP) and other machine-learning tasks, due to its unprecedented accuracy. However, their extensive memory and parameter requirements often hinder their practical applications. In this work, we study the effect of tensor-train decomposition to improve the accuracy and compress transformer vision-language neural networks, namely BERT and ViT. We focus both on embedding-layer compression and partial tensorization of neural networks (PTNN) through an algorithmic approach. Our novel PTNN approach significantly improves the accuracy of existing models by up to 5%, all without the need for post-training adjustments, breaking new ground in the field of tensor decomposition.

Compressing big data and model parameters via tensor decomposition such as the tensor train (TT) format has gained great success in recent years. The application of tensor compression methods requires the data be high dimensional. However, not all the real-world data primarily are high-dimensional, and sometimes reshaping is necessary before the application of tensor compression methods. Meantime, reordering and reshaping data may affect the efficiency of the compression. This work utilizes tensor reshaping to improve the efficiency of tensor compression using the TT format. An optimization model is proposed that maximizes the space-saving of tensor compression with respect to the shape of a given tensor while the compression error is bounded. The study is narrowed down to the TT decomposition and the TT-SVD algorithm is linked with a genetic algorithm (GA) to find an optimal tensor shape. The proposed method is applied to compress RGB images and a neural network to exemplify its capability. The results of the proposed tensor shape search using the GA are also compared with a purely random search. The results demonstrate that the proposed tensor shape search method significantly improves the space-saving and compression ratio of the data compression and enhances the efficiency of tensorized neural networks using the TT decomposition.

In this issue paper, the authors refine the definition of water sustainability to account for temporal dynamics and spatial variability, identify specific challenges that must be resolved in the very near future to avoid catastrophic outcomes on levels ranging from economic disruption to survival of mankind, discuss related policy changes and potential effectiveness, and describe several technologies available to achieve water security and sustainability. While water quality certainly poses formidable challenges, in this piece we emphasize and address challenges associated with dynamic water supply availability. Our future as a society will depend upon how well and how rapidly we navigate these challenges in the coming years. As such, the main objective is to encourage private and public sector practitioners to consider revising existing programs, and to update current industry business models in a manner that promotes expedited solutions, alignment of beneficial goals, and motivates the biggest consumers of water to adopt modern data collection and decision support technologies.

Various tensor decomposition methods have been proposed for data compression. In real world applications of the tensor decomposition, selecting the tensor shape for the given data poses a challenge and the shape of the tensor may affect the error and the compression ratio. In this work, we study the effect of the tensor shape on the tensor decomposition and propose an optimization model to find an optimum shape for the tensor train (TT) decomposition. The proposed optimization model maximizes the compression ratio of the TT decomposition given an error bound. We implement a genetic algorithm (GA) linked with the TT-SVD algorithm to solve the optimization model. We apply the proposed method for the compression of RGB images. The results demonstrate the effectiveness of the proposed evolutionary tensor shape search for the TT decomposition.

Hyper-spectral images are widely used for mapping and remote sensing of the Earth's surface. Different tensor decomposition methods have been applied for hyper-spectral image decomposition. In this study we present an evolutionary tensor train (ETT) decomposition. The ETT technique defines a combinatorial optimization model to find an optimal shape for the tensor train (TT) decomposition. The optimization model maximizes the compression ratio of the TT decomposition given an error bound. A genetic algorithm (GA) linked with the TT-SVD algorithm is applied to find the optimal shape. We adopt the ETT for the decomposition of hyper-spectral images and study the performance of the ETT with respect to the error bound. The results demonstrate the effectiveness of the proposed evolutionary tensor shape search for the the decomposition of the hyper-spectral images.

The application of neural networks (NN) in groundwater (GW) level prediction has been shown promising by previous works. Yet, previous works have relied on a variety of inputs, such as air temperature, pumping rates, precipitation, service population, and others. This work presents a long short-term memory neural network (LSTM-NN) for GW level forecasting using only previously observed GW level data as the input without resorting to any other type of data and information about a groundwater basin. This work applies the LSTM-NN for short-term and long-term GW level forecasting in the Edwards aquifer in Texas. The Adam optimizer is employed for training the LSTM-NN. The performance of the LSTM-NN was compared with that of a simple NN under 36 different scenarios with prediction horizons ranging from one day to three months, and covering several conditions of data availability. This paper’s results demonstrate the superiority of the LSTM-NN over the simple-NN in all scenarios and the success of the LSTM-NN in accurate GW level prediction. The LSTM-NN predicts one lag, up to four lags, and up to 26 lags ahead GW level with an accuracy (R2) of at least 99.89%, 99.00%, and 90.00%, respectively, over a testing period longer than 17 years of the most recent records. The quality of this work’s results demonstrates the capacity of machine learning (ML) in groundwater prediction, and affirms the importance of gathering high-quality, long-term, GW level data for predicting key groundwater characteristics useful in sustainable groundwater management.

Metaheuristics are widely applied to solve optimization problems. Numerous metaheuristic algorithms inspired by natural processes have been introduced in the past years. Studying and comparing the convergence of metaheuristics is helpful in future algorithmic development and applications. This study focuses on bee-inspired metaheuristics and identifies seven basic or root algorithms applied to solve continuous optimization problems. They are the bee system, mating bee optimization (MBO), bee colony optimization, bee evolution for genetic algorithms (BEGA), bee algorithm, artificial bee colony (ABC), and bee swarm optimization. The algorithms’ performances are evaluated with several benchmark problems. This study’s results rank the cited algorithms according to their convergence efficiency. The strengths and shortcomings of each algorithm are discussed. The ABC, BEGA, and MBO are the most efficient algorithms. This study’s results show the convergence rate among different algorithms varies, and evaluates the causes of such variation.