Narrow bit-width data formats are key to reducing the computational and storage costs of modern deep learning applications. This paper evaluates Microscaling (MX) data formats that combine a per-block scaling factor with narrow floating-point and integer types for individual elements. MX formats balance the competing needs of hardware efficiency, model accuracy, and user friction. Empirical results on over two dozen benchmarks demonstrate practicality of MX data formats as a drop-in replacement for baseline FP32 for AI inference and training with low user friction. We also show the first instance of training generative language models at sub-8-bit weights, activations, and gradients with minimal accuracy loss and no modifications to the training recipe.

This paper introduces Block Data Representations (BDR), a framework for exploring and evaluating a wide spectrum of narrow-precision formats for deep learning. It enables comparison of popular quantization standards, and through BDR, new formats based on shared microexponents (MX) are identified, which outperform other state-of-the-art quantization approaches, including narrow-precision floating-point and block floating-point. MX utilizes multiple levels of quantization scaling with ultra-fine scaling factors based on shared microexponents in the hardware. The effectiveness of MX is demonstrated on real-world models including large-scale generative pretraining and inferencing, and production-scale recommendation systems.

Automatic cranial model deformation has a significant impact on the ergonomic design of headgears. It benefits product design by providing accurate human cranial measurements while automatically deforming to target shapes. With the development of automatic deformation methods, cranial modeling can now be handled efficiently rather than manually customized. Furthermore, previous studies have shown that integrating anatomical landmarks in deformation methods can improve modeling accuracy. Hence, this study provides anatomical definitions of cranial landmarks, including 51 skull landmarks and 14 mandible landmarks.This study compares three different landmark-guided deformation methods using the above anatomic landmarks, including Landmark-Guided Coherent Point Drift (LGCPD), Neural Deformation Pyramid (NDP), and the registration part in SCULPTOR (S-ARAP). These three methods treat the automatic deformation problem as a task of probability density estimation, hierarchical deformation decomposition, and local rigidity preservation, respectively. However, LGCPD is computationally intensive, which means once the cranial model has many vertices, the computation consumes a large memory and runs slowly. Additionally, LGCPD is sensitive to obtain a suboptimal solution and results in a deformed model with a high shape variation. NDP simplifies the deformation problem by decomposing it into several sub-deformations. With Multi-Layer Perception (MLP), NDP can perform the deformation approximately 10 times faster than LGCPD. However, without the constraint of local rigidity, partial-to-partial deformation accumulates minor deformation errors from each sub-step, leading to unsatisfactory deformation results. S-ARAP uniformly samples control nodes and computes their influence weights on the source model's vertices using Radial Basis Function (RBF). The larger the distance between the node and the vertices, the higher the weight with a stronger influence. The as-rigid-as-possible (ARAP) term is then introduced to preserve the local rigidity of the deformed model with the calculated influence weights for the local regions. Therefore, S-ARAP can automatically deform the cranial model, particularly the skull part with complex geometries, to achieve a well-structured result. Moreover, the control node sampling speeds up the execution of deformation while using less memory than LGCPD. Instead of the decomposition in NDP, S-ARAP increases the number of control nodes in several stages to perform hierarchical deformation.Finally, with quantitative and qualitative experimental results, the study discusses and compares the suitability of these three deformation methods for automatic cranial modeling. The study computes Chamfer-Distance (CD) and Point-to-Plane Distance (PTPD) on the deformed results for quantitative comparisons. CD determines the distance between deformed vertices and the nearest vertices on the target model and vice versa. PTPD calculates the distance between the deformed vertices and the nearest plane on the target model to calculate the shape error. The maximum value in PTPD can help identify outliers in deformed results. Lower CD and PTPD values suggest a better match with the target. According to the experimental results, S-ARAP outperforms LGCPD and NDP in terms of CD and PTPD. Furthermore, the deformed data are visualized with a heatmap revealing the large deformation error, and S-ARAP shows the lowest fitting error on the deformed results. Thus, S-ARAP is a suitable method for automatic deformation on cranial modeling.

Una primera tarjeta de interfaz de red en linea, NIC (104), para indexar los flujos de red (106), comprendiendo la primera NIC en linea (104): un primer controlador de acceso a los medios, MAC (150); un segundo MAC (152); hardware de procesamiento (124) configurado para proporcionar la transmision de paso (154, 156) de los paquetes de los flujos de red (106) mediante la transmision de los paquetes del primer MAC (150) recibidos por el segundo MAC (152) y mediante la transmision de los paquetes del segundo MAC (152) recibidos por el primer MAC (150); un primer modulo (130) configurado para implementar un protocolo de transporte ligero, LTP; y un segundo modulo (132) configurado para comunicarse con una segunda NIC en linea arbitraria por medio del primer modulo (130) especificando una direccion de red correspondiente a la segunda NIC en linea para permitir que el primer modulo (130) establezca una conexion LTP (108) con puntos finales en la primera NIC en linea (104) y en la segunda NIC en linea, en donde la primera NIC en linea (104) se conecta a un primer anfitrion y a la red de datos, y la segunda NIC en linea se conecta a un segundo anfitrion y a la red de datos, en donde el hardware de procesamiento se configura para proporcionar conectividad de red entre el primer anfitrion y el segundo anfitrion, realizando una transmision de paso de los paquetes recibidos que se ha determinado que no son paquetes LTP entre NIC, y proporcionar conectividad LTP entre NIC entre la primera NIC en linea (104) y la segunda NIC en linea, y en donde los paquetes recibidos por la primera NIC en linea que se determina que son paquetes LTP entre IC se consumen por la primera NIC en linea (104) y no se reenvian al primer o segundo anfitrion mediante la primera NIC en linea, y en donde los paquetes LTP entre NIC se originan por la primera o segunda NIC en linea.

Un metodo para restaurar la aceleracion del servicio para un servicio, el metodo que comprende: determinar que la aceleracion del servicio para el servicio esta operando incorrectamente, la aceleracion del servicio proporcionada por un grupo de componentes de aceleracion de interoperacion (1301, 1302, 1303, 1304) y por papeles (1311, 1312, 1313, 1314) en cada componente de aceleracion en el grupo de componentes de aceleracion de interoperacion enlazados entre si para componer un grafico (1333), en donde los componentes de aceleracion estan en un plano de aceleracion de hardware; detectar que el rendimiento degradado en un componente de aceleracion (1303), incluido en el grupo de componentes de aceleracion de interoperacion, hizo que la aceleracion del servicio operase incorrectamente, el componente de aceleracion asignado para proporcionar un papel (1313) que esta enlazado a uno o mas de otros papeles (1312, 1314) en el grafico; seleccionar un componente de aceleracion de sustitucion (1306) de entre uno o mas de otros componentes de aceleracion para proporcionar el papel; y restaurar la aceleracion del servicio para el servicio asignando el componente de aceleracion de sustitucion para proporcionar el papel y enlazar el papel (1313) proporcionado por el componente de aceleracion de sustitucion con uno o mas de otros papeles (1312, 1314).

Apparatus and methods for training neural networks based on a performance metric, including adjusting numerical precision and topology as training progresses are disclosed. In some examples, block floating-point formats having relatively lower accuracy are used during early stages of training. Accuracy of the floating-point format can be increased as training progresses based on a determined performance metric. In some examples, values for the neural network are transformed to normal precision floating-point formats. The performance metric can be determined based on entropy of values for the neural network, accuracy of the neural network, or by other suitable techniques. Accelerator hardware can be used to implement certain implementations, including hardware having direct support for block floating-point formats.

A system for block floating point computation in a neural network receives a block floating point number comprising a mantissa portion. A bit-width of the block floating point number is reduced by decomposing the block floating point number into a plurality of numbers each having a mantissa portion with a bit-width that is smaller than a bit-width of the mantissa portion of the block floating point number. One or more dot product operations are performed separately on each of the plurality of numbers to obtain individual results, which are summed to generate a final dot product value. The final dot product value is used to implement the neural network. The reduced bit width computations allow higher precision mathematical operations to be performed on lower-precision processors with improved accuracy.

Perplexity scores are computed for training data samples during ANN training. Perplexity scores can be computed as a divergence between data defining a class associated with a current training data sample and a probability vector generated by the ANN model. Perplexity scores can alternately be computed by learning a probability density function ("PDF") fitting activation maps generated by an ANN model during training. A perplexity score can then be computed for a current training data sample by computing a probability for the current training data sample based on the PDF. If the perplexity score for a training data sample is lower than a threshold, the training data sample is removed from the training data set so that it will not be utilized for training during subsequent epochs. Training of the ANN model continues following the removal of training data samples from the training data set.

Technology related to training a neural network accelerator using mixed precision data formats is disclosed. In one example of the disclosed technology, a neural network accelerator is configured to accelerate a given layer of a multi-layer neural network. An input tensor for the given layer can be converted from a normal-precision floating-point format to a quantized-precision floating-point format. A tensor operation can be performed using the converted input tensor. A result of the tensor operation can be converted from the block floating-point format to the normal-precision floating-point format. The converted result can be used to generate an output tensor of the layer of the neural network, where the output tensor is in normal-precision floating-point format.

In this paper, we explore the limits of Microsoft Floating Point (MSFP), a new class of datatypes developed for production cloud-scale inferencing on custom hardware. Through the co-evolution of hardware design and algorithms, MSFP achieves accuracy comparable to or better than industry standards Bfloat16 and INT8 at 3x and 4x lower cost, respectively. MSFP incurs negligible impact to accuracy (

Methods and apparatus are disclosed for providing emulation of quantized precision operations in a neural network. In some examples, the quantized precision operations are performed in a block floating-point format where values of a tensor share a common exponent. Techniques for selecting higher precision or lower precision can be used based on a variety of input metrics. When converting to a quantized tensor, a residual tensor is produced. In one embodiment, an error value associated with converting from a normal-precision floating point number to the quantized tensor is used to determine whether to use the residual tensor in a dot product calculation. Using the residual tensor increases the precision of an output from a node. Selection of whether to use the residual tensor can depend on various input metrics including the error value, the layer number, the exponent value, the layer type, etc.

Technology related to incremental training of machine learning tools is disclosed. In one example of the disclosed technology, a method can include receiving operational parameters of a machine learning tool based on a primary set of training data. The machine learning tool can be a deep neural network. Input data can be applied to the machine learning tool to generate an output of the machine learning tool. A measure of prediction quality can be generated for the output of the machine learning tool. In response to determining the measure of prediction quality is below a threshold, incremental training of the operational parameters can be initiated using the input data as training data for the machine learning tool. Operational parameters of the machine learning tool can be updated based on the incremental training. The updated operational parameters can be stored.

Growing computational demands from deep neural networks (DNNs), coupled with diminishing returns from general-purpose architectures, have led to a proliferation of Neural Processing Units (NPUs). This paper describes the Project Brainwave NPU (BW-NPU), a parameterized microarchitecture specialized at synthesis time for convolutional and recurrent DNN workloads. The BW-NPU deployed on an Intel Stratix 10 280 FPGA achieves sustained performance of 35 teraflops at a batch size of 1 on a large recurrent neural network (RNN).

Methods and apparatus are disclosed supporting a design flow for developing quantized neural networks. In one example of the disclosed technology, a method includes quantizing a normal-precision floating-point neural network model into a quantized format. For example, the quantized format can be a block floating-point format, where two or more elements of tensors in the neural network share a common exponent. A set of test input is applied to a normal-precision flooding point model and the corresponding quantized model and the respective output tensors are compared. Based on this comparison, hyperparameters or other attributes of the neural networks can be adjusted. Further, quantization parameters determining the widths of data and selection of shared exponents for the block floating-point format can be selected. An adjusted, quantized neural network is retrained and programmed into a hardware accelerator.

A server system is provided that includes a plurality of servers, each server including at least one hardware acceleration device and at least one processor communicatively coupled to the hardware acceleration device by an internal data bus and executing a host server instance, the host server instances of the plurality of servers collectively providing a software plane, and the hardware acceleration devices of the plurality of servers collectively providing a hardware acceleration plane that implements a plurality of hardware accelerated services, wherein each hardware acceleration device maintains in memory a data structure that contains load data indicating a load of each of a plurality of target hardware acceleration devices, and wherein a requesting hardware acceleration device routes the request to a target hardware acceleration device that is indicated by the load data in the data structure to have a lower load than other of the target hardware acceleration devices.

A method is provided for implementing a deep neural network on a server component that includes a host component including a CPU and a hardware acceleration component coupled to the host component. The deep neural network includes a plurality of layers. The method includes partitioning the deep neural network into a first segment and a second segment, the first segment including a first subset of the plurality of layers, the second segment including a second subset of the plurality of layers, configuring the host component to implement the first segment, and configuring the hardware acceleration component to implement the second segment.

Technology related to hardware accelerated neural network subgraphs is disclosed. In one example of the disclosed technology, a method for compiling a neural network model is disclosed. The method includes identifying a subgraph of the neural network model to partition from the neural network model. An interface can be inserted between the neural network model and a partitioned version of the identified subgraph. The partitioned version can be adapted to be evaluated with a neural network accelerator. The identified subgraph can be compiled to the neural network accelerator to generate configuration information for the neural network accelerator. The neural network accelerator can be configured with the configuration information to provide an accelerated version of the subgraph.

A smart NIC (Network Interface Card) is provided with features to enable the smart NIC to operate as an in-line NIC between a host's NIC and a network. The smart NIC provides pass-through transmission of network flows for the host. Packets sent to and from the host pass through the smart NIC. As a pass-through point, the smart NIC is able to accelerate the performance of the pass-through network flows by analyzing packets, inserting packets, dropping packets, inserting or recognizing congestion information, and so forth. In addition, the smart NIC provides a lightweight transport protocol (LTP) module that enables it to establish connections with other smart NICs. The LTP connections allow the smart NICs to exchange data without passing network traffic through their respective hosts.

Tensor register files in a hardware accelerator are disclosed. An apparatus may comprise tensor operation calculators each configured to perform a type of tensor operation. The apparatus may also comprises tensor register files, each of which is associated with one of the tensor operation calculators. The apparatus may also comprises logic configured to store respective ones of the tensors in the plurality of tensor register files in accordance with the type of tensor operation to be performed on the respective tensors. The apparatus may also control read access to tensor register files based on a type of tensor operation that a machine instruction is to perform.