Speculative decoding has emerged as a powerful method to improve latency and throughput in hosting large language models. However, most existing implementations focus on generating a single sequence. Real-world generative AI applications often require multiple responses and how to perform speculative decoding in a batched setting while preserving its latency benefits poses non-trivial challenges. This paper describes a system of batched speculative decoding that sets a new state of the art in multi-sequence generation latency and that demonstrates superior GPU utilization as well as quality of generations within a time budget. For example, for a 7.8B-size model on a single A100 GPU and with a batch size of 8, each sequence is generated at an average speed of 5.8ms per token, the overall throughput being 1.1K tokens per second. These results represent state-of-the-art latency and a 2.15X speed-up over optimized regular decoding. Within a time budget that regular decoding does not finish, our system is able to generate sequences with HumanEval Pass@First of 43% and Pass@All of 61%, far exceeding what's feasible with single-sequence speculative decoding. Our peak GPU utilization during decoding reaches as high as 15.8%, more than 3X the highest of that of regular decoding and around 10X of single-sequence speculative decoding.

This study introduces bifurcated attention, a method designed to enhance language model inference in shared-context batch decoding scenarios. Our approach addresses the challenge of redundant memory IO costs, a critical factor contributing to latency in high batch sizes and extended context lengths. Bifurcated attention achieves this by strategically dividing the attention mechanism during incremental decoding into two separate GEMM operations: one focusing on the KV cache from prefill, and another on the decoding process itself. While maintaining the computational load (FLOPs) of standard attention mechanisms, bifurcated attention ensures precise computation with significantly reduced memory IO. Our empirical results show over 2.1× speedup when sampling 16 output sequences and more than 6.2× speedup when sampling 32 sequences at context lengths exceeding 8k tokens on a 7B model that uses multi-head attention. The efficiency gains from bifurcated attention translate into lower latency, making it particularly suitable for real-time applications. For instance, it enables massively parallel answer generation without substantially increasing latency, thus enhancing performance when integrated with post-processing techniques such as re-ranking.

In our study, we present bifurcated attention, a method developed for language model inference in single-context batch sampling contexts. This approach aims to reduce redundant memory IO costs, a significant factor in latency for high batch sizes and long context lengths. Bifurcated attention achieves this by dividing the attention mechanism during incremental decoding into two distinct GEMM operations, focusing on the KV cache from prefill and the decoding process. This method ensures precise computation and maintains the usual computational load (FLOPs) of standard attention mechanisms, but with reduced memory IO. Bifurcated attention is also compatible with multi-query attention mechanism known for reduced memory IO for KV cache, further enabling higher batch size and context length. The resulting efficiency leads to lower latency, improving suitability for real-time applications, e.g., enabling massively-parallel answer generation without substantially increasing latency, enhancing performance when integrated with postprocessing techniques such as reranking.

As modern hardware architectures evolve to support increasingly diverse, complex instruction sets for meeting the performance demands of modern workloads in image processing, deep learning, etc., it has become ever more crucial for compilers to provide robust support for evolution of their internal abstractions and retargetable code generation support to keep pace with emerging instruction sets. We propose Hydride, a novel approach to compiling for complex, emerging hardware architectures. Hydride uses vendor-defined pseudocode specifications of multiple hardware ISAs to automatically design retargetable instructions for AutoLLVM IR, an extensible compiler IR which consists of (formally defined) language-independent and target-independent LLVM IR instructions to compile to those ISAs, and automatically generated instruction selection passes to lower AutoLLVM IR to each of the specified hardware ISAs. Hydride also includes a code synthesizer that automatically generates code generation support for schedule-based languages, such as Halide, to optimally generate AutoLLVM IR. Our results show that Hydride is able to represent 3,557 instructions combined in x86, Hexagon, ARM architectures using only 397 AutoLLVM IR instructions, including (Intel) SSE2, SSE4, AVX, AVX2, AVX512, (Qualcomm) Hexagon HVX, and (ARM) NEON vector ISAs. We created a new Halide compiler with Hydride using only a formal semantics of Halide IR, leveraging the auto-generated AutoLLVM IR and back-ends for the three hardware architectures. Across kernels from deep learning and image processing, this compiler is able to perform just as well as the mature, production Halide compiler on Hexagon, and outperform on x86 by 8% and ARM by 3%. Hydride also outperforms the production Halide's LLVM back end by 12% on x86, 100% on HVX, and 26% on ARM across the same kernels.

Large language models trained on code have shown great potential to increase productivity of software developers. Several execution-based benchmarks have been proposed to evaluate functional correctness of model-generated code on simple programming problems. Nevertheless, it is expensive to perform the same evaluation on complex real-world projects considering the execution cost. On the contrary, static analysis tools such as linters, which can detect errors without running the program, haven't been well explored for evaluating code generation models. In this work, we propose a static evaluation framework to quantify static errors in Python code completions, by leveraging Abstract Syntax Trees. Compared with execution-based evaluation, our method is not only more efficient, but also applicable to code in the wild. For experiments, we collect code context from open source repos to generate one million function bodies using public models. Our static analysis reveals that Undefined Name and Unused Variable are the most common errors among others made by language models. Through extensive studies, we also show the impact of sampling temperature, model size, and context on static errors in code completions.

We present new benchmarks on evaluation code generation models: MBXP and Multilingual HumanEval, and MathQA-X. These datasets cover over 10 programming languages and are generated using a scalable conversion framework that transpiles prompts and test cases from the original Python datasets into the corresponding data in the target language. Using these benchmarks, we are able to assess the performance of code generation models in a multi-lingual fashion, and discovered generalization ability of language models on out-of-domain languages, advantages of multi-lingual models over mono-lingual, the ability of few-shot prompting to teach the model new languages, and zero-shot translation abilities even on mono-lingual settings. Furthermore, we use our code generation model to perform large-scale bootstrapping to obtain synthetic canonical solutions in several languages, which can be used for other code-related evaluations such as code insertion, robustness, or summarization tasks. Overall, our benchmarks represents a significant step towards a deeper understanding of language models' code generation abilities. We publicly release our code and datasets at https://github.com/amazon-research/mxeval.

The Precambrian basement of the Schirmacher Hills records multiple episodes of deformation, metamorphism, migmatization, and emplacement of successive generations of mafic and felsic bodies. The earliest tectono-thermal event (D1/M1), preserved in some mafic and ultramafic enclaves, indicates deformation at great crustal depth. The mineralogical assemblage of these enclaves indicates early high temperature (900o C) and high-pressure (10 Kbar) granulite facies conditions. The second tectono-thermal event also showed deformation under granulite facies metamorphism (D2/M2) under 800-850oC and 8 Kbar. The third group of events (D3/M3) is the most dominant in this region and involved deformation under amphibolite facies conditions with synchronous emplacement of granites and mafic dykes and culminated in regional thrusting, producing a regional inversion where the granulates were emplaced over the amphibolite facies rocks. The later events created upright folds and vertical shear zones under amphibolite facies conditions.

Most recent research on Text-to-SQL semantic parsing relies on either parser itself or simple heuristic based approach to understand natural language query (NLQ). When synthesizing a SQL query, there is no explicit semantic information of NLQ available to the parser which leads to undesirable generalization performance. In addition, without lexical-level fine-grained query understanding, linking between query and database can only rely on fuzzy string match which leads to suboptimal performance in real applications. In view of this, in this paper we present a general-purpose, modular neural semantic parsing framework that is based on token-level fine-grained query understanding. Our framework consists of three modules: named entity recognizer (NER), neural entity linker (NEL) and neural semantic parser (NSP). By jointly modeling query and database, NER model analyzes user intents and identifies entities in the query. NEL model links typed entities to schema and cell values in database. Parser model leverages available semantic information and linking results and synthesizes tree-structured SQL queries based on dynamically generated grammar. Experiments on SQUALL, a newly released semantic parsing dataset, show that we can achieve 56.8% execution accuracy on WikiTableQuestions (WTQ) test set, which outperforms the state-of-the-art model by 2.7%.

Retrieval-based language models (R-LM) model the probability of natural language text by combining a standard language model (LM) with examples retrieved from an external datastore at test time. While effective, a major bottleneck of using these models in practice is the computationally costly datastore search, which can be performed as frequently as every time step. In this paper, we present RetoMaton - retrieval automaton - which approximates the datastore search, based on (1) saving pointers between consecutive datastore entries, and (2) clustering of entries into "states". This effectively results in a weighted finite automaton built on top of the datastore, instead of representing the datastore as a flat list. The creation of the automaton is unsupervised, and a RetoMaton can be constructed from any text collection: either the original training corpus or from another domain. Traversing this automaton at inference time, in parallel to the LM inference, reduces its perplexity by up to 1.85, or alternatively saves up to 83% of the nearest neighbor searches over $k$NN-LM (Khandelwal et al., 2020) without hurting perplexity. Our code and trained models are available at https://github.com/neulab/retomaton .

Geotagged tweet streams contain invaluable information about the real-world local events like sports games, protests and traffic accidents. Timely detecting and extracting such events may have various applications but yet unsolved challenges. In this paper, we present DeLLe, a methodology for automatically Detecting Latest Local Events from geotagged tweets. With the help of novel spatio temporal tweet count prediction models, DeLLe first finds unusual locations which have aggregated unexpected number of tweets in the latest time period and thereby imply potential local events. Next, DeLLe calculates, for each such unusual location, a ranking score to identify the ones most likely having ongoing local events by addressing the temporal burstiness, spatial burstiness and topical coherence. Furthermore, DeLLe infers an event candidate's spatio temporal range by tracking its event-focus point, which essentially reflects the most recent representative occurrence site. Finally, DeLLe chooses the most influential tweets to summarize local events and thereby presents succinct but yet representative descriptions. We evaluate DeLLe on the city of Seattle, WA as well as a larger city of New York. The results show that the proposed method generally outperforms competitive baseline approaches.

Programmable software-defined solid-state drives can move computing functions closer to storage.

The tweet count prediction of a local spatial region is to forecast the number of tweets that are likely to be posted from that area over a relatively short period of time. It has many applications such as human mobility analysis, traffic planning, and abnormal event detection. In this paper, we formulate tweet count prediction as a spatiotemporal sequence forecasting problem and design an end-to-end convolutional LSTM based network with skip connection for this problem. Such a model enables us to exploit the unique properties of spatiotemporal data, consisting of not only the temporal characteristics such as temporal closeness, period and trend properties but also spatial dependencies. Our experiments on the city of Seattle, WA as well as a larger city of New York City show that the proposed method consistently outperforms the competitive baseline approaches.

Geotagged tweet streams contain invaluable information about the real-world local events like sports games, protests and traffic accidents. Timely detecting and extracting such events has various applications but yet unsolved challenges. In this paper, we present DeLLe, a methodology for automatically Detecting Latest Local Events from geotagged tweet streams. DeLLe first finds unusual locations which have aggregated unexpected number of tweets, and then ranks the unusual locations to select the top ones that are likely to be local event candidates. We evaluate DeLLe on the city of Seattle, WA as well as a larger city of New York. The results show that the proposed method generally outperforms competitive baseline approaches.

dbDedup is a similarity-based deduplication scheme for on-line database management systems (DBMSs). Beyond block-level compression of individual database pages or operation log (oplog) messages, as used in today's DBMSs, dbDedup uses byte-level delta encoding of individual records within the database to achieve greater savings. dbDedup's single-pass encoding method can be integrated into the storage and logging components of a DBMS to provide two benefits: (1) reduced size of data stored on disk beyond what traditional compression schemes provide, and (2) reduced amount of data transmitted over the network for replication services. To evaluate our work, we implemented dbDedup in a distributed NoSQL DBMS and analyzed its properties using four real datasets. Our results show that dbDedup achieves up to 37 x reduction in the storage size and replication traffic of the database on its own and up to 61 x reduction when paired with the DBMS's block-level compression. dbDedup provides both benefits with negligible effect on DBMS throughput or client latency (average and tail).

Un sistema que comprende un servicio (116) de indice hash que comprende un dispositivo (124) de almacenamiento primario y un dispositivo (128) de almacenamiento secundario, en el que: el dispositivo de almacenamiento secundario comprende un indice (126) hash estructurado en registros mantenido en el dispositivo de almacenamiento secundario, en el que las entradas del indice hash estructurado en registros comprenden valores hash de fragmentos de datos y metadatos asociados con cada valor hash; el dispositivo de almacenamiento primario comprende una tabla (122) de indices compactos que incluye las firmas compactas representativas de los valores hash en el indice hash, y para cada firma compacta, un puntero a una localizacion del valor hash correspondiente en el indice hash; y el servicio de indice hash se configura para acceder a la tabla de indices compactos para buscar una firma compacta que corresponde a un valor hash solicitado para buscar, y para devolver un resultado de no encontrado, si la firma compacta no se encuentra en la tabla de indices compactos, y para seguir el puntero para buscar una entrada en el indice hash estructurado en registros que posiblemente contiene el valor hash solicitado si la firma compacta se encuentra en la tabla de indices compactos.

dDedup is a similarity-based deduplication scheme for on-line database management systems (DBMSs). Beyond block-level compression of individual database pages or operation log (oplog) messages, as used in today’s DBMSs, dDedup uses byte-level delta encoding of individual records within the database to achieve greater savings. dDedup’s single-pass encoding method can be integrated into the storage and logging components of a DBMS to provide two benefits: (1) reduced size of data stored on disk beyond what traditional compression schemes provide, and (2) reduced amount of data transmitted over the network for replication services. To evaluate our work, we implemented dDedup in a distributed NoSQL DBMS and analyzed its properties using four real datasets. Our results show that dDedup achieves up to 37X reduction in the storage size and replication traffic of the database on its own and up to 61X reduction when paired with the DBMS’s block-level compression. dDedup provides both benefits with negligible effect on DBMS throughput or client latency (average and tail).

A longstanding goal of SSD virtualization has been to provide performance isolation between multiple tenants sharing the device. Virtualizing SSDs, however, has traditionally been a challenge because of the fundamental tussle between resource isolation and the lifetime of the device - existing SSDs aim to uniformly age all the regions of flash and this hurts isolation. We propose utilizing flash parallelism to improve isolation between virtual SSDs by running them on dedicated channels and dies. Furthermore, we offer a complete solution by also managing the wear. We propose allowing the wear of different channels and dies to diverge at fine time granularities in favor of isolation and adjusting that imbalance at a coarse time granularity in a principled manner. Our experiments show that the new SSD wears uniformly while the 99th percentile latencies of storage operations in a variety of multi-tenant settings are reduced by up to 3.1x compared to software isolated virtual SSDs.

The Bw-tree is a latch-free, B-tree style index that is part of Deuteronomy key-value stores, which have been deployed both in the cloud and in “boxed” products. It can be layered on a log structured storage manager or used as a main memory index, and it can support transactions when embedded in a system with a Deuteronomy transactional component. Its high performance is a result of its latch-free, log structured properties, coupled with an intelligent but straightforward implementation of index search. In this paper, we present our optimized Bw-tree index that is substantially faster than our initial implementation. We apply an integrated set optimizations, new ones and refinements of prior techniques, to produce Bw-tree indexing with up to 40 percent better performance for single key lookups, while also improving range search performance. And our original implementation performs comparably to the best main memory indexes. Uniquely, however, it is designed to be used with data residing on secondary storage, and hence includes the overheads required to keep data paginated.

Phulad Shear Zone (PSZ) of Delhi Fold Belt in Rajasthan is a northeasterly striking ductile shear zone with a well developed mylonitic foliation (035/70E) and a downdip stretching lineation. The deformation in the PSZ has developed in a transpressional regime with thrusting sense of movement. The northeastern unit, i.e., the hanging wall contains a variety of rocks namely calc-silicates, pelites and amphibolites and the southwestern unit, i.e., the footwall unit contains only granitic rocks. Systematic investigation of the granites of the southwestern unit indicate a gradual change in the intensity of deformation from a distance of about 1 km west of the shear zone to the shear zone proper. The granite changes from weakly deformed granite to a mylonite/ultramylonite as we proceed towards the PSZ. The weakly deformed granite shows a crude foliation with the same attitude of mylonitic foliation of the PSZ. Microscopic study reveals the incipient development of C and S fabric with angle between C and S varying from 15 ∘ to 24 ∘. The small angle between the C and S fabric in the least deformed granite variety indicates that the deformation has strong pure shear component. At a distance of about 1 m away from the PSZ, there is abrupt change in the intensity of deformation. The granite becomes intensely foliated with a strong downdip lineation and the rock becomes a true mylonite. In mesoscopic scale, the granite shows stretched porphyroclasts in both XZ and YZ sections indicating a flattening type of deformation. The angle between the C and S fabric is further reduced and finally becomes nearly parallel. In most places, S fabric is gradually replaced by C fabric. Calculation of sectional kinematic vorticity number ( W n) from the protomylonitic and mylonite/ultramylonite granites varies from 0.3 ± 0.03 to 0.55 ± 0.04 indicating a strong component of pure shear. The similarity of the geometry of structures in the PSZ and the granites demonstrates that the deformation of the two units is broadly synchronous and the deformation in both the units is transpressional.