Modular adders are essential arithmetic components in Residue Number System (RNS)-based applications, including digital signal processing, cryptography, and machine learning. These applications consistently push the boundaries of dynamic range (DR) and operating frequency, making the design of efficient generic modular adders a critical and evolving challenge. This paper presents a novel algorithm for modulo-(2(n) +/-delta) addition, where.. is an integer within the range 0 <= delta <= 2(n-1) The proposed approach leverages a two-valued digit (twit) for encoding the value of +/-delta and uses a faithful representation of operands. In this representation, each operand is encoded as an..-bit unsigned number augmented by a twit value {0, +/- delta}. The algorithm efficiently performs modular addition by speculating and adjusting the twit value in the addition result. When the result exceeds the modulus, it subtracts 2(n) +/- delta by ignoring the carry-out and adjusting the speculated twit value. This adjustment is achieved through an XOR operation between the carry-out and the speculated twit value, simplifying the modular reduction process. The proposed design has been synthesized for practical.. (4 <= n <= 16) using a FreePDK 45 nm process. The results demonstrate superior performance across key metrics such as delay, area, and power consumption compared to previous designs, highlighting the efficacy and scalability of the approach.
We define flap to mean a floating point amalgamated operation, i.e., an add, multiply, or multiply-add operation, and derive asymptotic flop and flap counts for the radix-2, radix-4, radix-8, 2,4-split-radix, and 2,8-split-radix FFT algorithms. We find that, of these, the 2,8-split-radix algorithm requires the fewest flaps. Lastly, we opine on real-value scaling of an FFT.
This paper presents a comprehensive evaluation of microscaling (MX) quantization in the pre-training of large language models (LLMs), investigating its potential to enhance the computation and memory efficiencies. We systematically examine the effects of key design parameters - including data types, rounding modes, scaling strategies, granularity, and organization - on numerical accuracy and training stability. Our extensive experimental study on Llama3 models reveals critical insights into the challenges of 4-bit training for LLMs and identifies optimal configurations with mixed precisions of 4-bit and 6-bit MX formats that significantly enhance training quality, bridging the gap with higher-precision formats. This research provides valuable guidance on the benefits and limitations of MX quantization, laying the groundwork for future innovations in low-precision LLM training.
Floating-point fused multiply-add (FMA) operations are fundamental in many fields such as scientific computing, graphics processing, and machine learning. In conventional floating-point FMA designs, the internal steps of multiplication and addition, which are performed using integer arithmetic, are followed by a post-processing stage. It has been known that the post-processing stage contributes to approximately 60% of the total latency of the double-precision FMA operation. We propose a novel trailing-ones anticipation technique that predicts trailing-ones bits of the mantissa before rounding, in parallel with the post-processing stage. With this technique, the rounding incrementer can be implemented using a single XOR operation, thereby reducing the total latency. We evaluated the latency using Synopsys Design Compiler for synthesis, confirming that the proposed technique reduced total latency by 4 %.
To date, the most promising methods for 8-bit DNN training use two different floating-point formats: a 5-bit ex-ponent for greater range on gradients in the backwards pass, and a 4-bit exponent with less range but greater precision for representing weights and activations in the forward pass. Ordinarily, subnormal representations are also used to extend the range and provide gradual underflow, but subnormals require considerable special-case logic which increases the cost of area-critical operators such as multiply-accumulate units. Instead, this paper proposes to replace traditional subnormals with what we call supernormal values where mantissa bits are converted into exponent bits at both the subnormal/zero and infinity ends of the range. Hence, supernormals provide maximal range extension by sacrificing some precision at both ends. The logic required to implement supernormals is a trivial change to existing logic that processes normalised values, thus avoiding the logic overhead of subnormals while providing a larger range extension. We use the range extension provided by supernormals in new mixed-precision 8-bit formats to showcase training results that are comparable to the state-of-the-art while showing an area savings.
