We show that DNN accelerator micro-architectures and their program mappings represent specific choices of loop order and hardware parallelism for computing the seven nested loops of DNNs, which enables us to create a formal taxonomy of all existing dense DNN accelerators. Surprisingly, the loop transformations needed to create these hardware variants can be precisely and concisely represented by Halide's scheduling language. By modifying the Halide compiler to generate hardware, we create a system that can fairly compare these prior accelerators. As long as proper loop blocking schemes are used, and the hardware can support mapping replicated loops, many different hardware dataflows yield similar energy efficiency with good performance. This is because the loop blocking can ensure that most data references stay on-chip with good locality and the processing units have high resource utilization. How resources are allocated, especially in the memory system, has a large impact on energy and performance. By optimizing hardware resource allocation while keeping throughput constant, we achieve up to 4.2X energy improvement for Convolutional Neural Networks (CNNs), 1.6X and 1.8X improvement for Long Short-Term Memories (LSTMs) and multi-layer perceptrons (MLPs), respectively.

Most emerging applications in imaging and machine learning must perform immense amounts of computation while holding to strict limits on energy and power. To meet these goals, architects are building

Student motivation is critical to learning and program retention in engineering, yet most introductory circuits classes present the material in an abstract manner that does little to inspire students. In Stanford's introductory circuits course, 58% of the students are not electrical engineering majors, and generally have little intrinsic motivation for learning circuits, since it is not their chosen field. Another 39% are undeclared, and using the course to get a feel for electrical engineering as a whole. Our traditional linear circuits class covers Kirchhoff’s laws, nodal analysis, Thévenin/Norton, first-order response in the time domain for RL and RC circuits, op-amps, and phasors. Although this provides an important foundation for electrical engineers, it is not neither helpful nor interesting to non-majors who will not take further circuits classes. Moreover, it conveys a very narrow view of EE for those scouting out the major. Four years ago, we completely redesigned our introductory circuits class to address these shortcomings. The new course is focused around a sequence of fun and practical lab projects, where students use what they are learning to build complete devices: a solar-powered cell phone charger, a trick box which turns itself off, an audio-controlled LED cube, and an electrocardiogram. The lectures explain the material just in time for each lab, beginning with linear circuits, but quickly detouring to explore diodes, solar cells, transistors, digital logic, and a bit of microcontroller programming. We return to linear circuits after introducing the frequency domain, and discuss filters and amplifiers. We explicitly teach students techniques for building physical circuits and devices, and labs are graded for quality of design and construction in addition to electrical functionality. This broader range of theoretical topics and practical skills provides students (especially non-majors) with a more powerful toolbox for building useful circuits. Realistic applications are further emphasized through the homework, exams, and a series of in-class "breaking breaks". There were several positive results after introducing the new course. We experienced a large demographic shift in the course, with students taking the class earlier in their career and before declaring a major. Student evaluations of the labs have been consistently positive, and a handful of students specifically cited the course as their reason for choosing to major in electrical engineering.

Many DNN accelerators have been proposed and built using different microarchitectures and program mappings. To fairly compare these different approaches, we modified the Halide compiler to produce hardware as well as CPU and GPU code, and show that Halide's existing scheduling language has enough power to represent all existing dense DNN accelerators. Using this system we can show that the specific dataflow chosen for the accelerator is not critical to achieve good efficiency: many different dataflows yield similar energy efficiency with good performance. However, finding the best blocking and resource allocation is critical, and we achieve a 2.6X energy savings over Eyeriss system by reducing the size of the local register file. Adding an additional level in the memory hierarchy saves an additional 25%. Based on these observations, we develop an optimizer that automatically finds the optimal blocking and storage hierarchy. Compared with Eyeriss system, it achieves up to 4.2X energy improvement for Convolutional Neural Networks (CNNs), 1.6X and 1.8X improvement for Long Short-Term Memories (LSTMs) and multi-layer perceptrons (MLPs) respectively.

Specialized image processing accelerators are necessary to deliver the performance and energy efficiency required by important applications in computer vision, computational photography, and augmented reality. But creating, “programming,” and integrating this hardware into a hardware/software system is difficult. We address this problem by extending the image processing language Halide so users can specify which portions of their applications should become hardware accelerators, and then we provide a compiler that uses this code to automatically create the accelerator along with the “glue” code needed for the user’s application to access this hardware. Starting with Halide not only provides a very high-level functional description of the hardware but also allows our compiler to generate a complete software application, which accesses the hardware for acceleration when appropriate. Our system also provides high-level semantics to explore different mappings of applications to a heterogeneous system, including the flexibility of being able to change the throughput rate of the generated hardware. We demonstrate our approach by mapping applications to a commercial Xilinx Zynq system. Using its FPGA with two low-power ARM cores, our design achieves up to 6× higher performance and 38× lower energy compared to the quad-core ARM CPU on an NVIDIA Tegra K1, and 3.5× higher performance with 12× lower energy compared to the K1’s 192-core GPU.

We present a method for video stabilization and rolling-shutter correction for videos captured on mobile devices. The method uses the data from an on-board gyroscope to track the camera's angular velocity, and can run in real time within the camera capture pipeline. We remove small motions and rolling-shutter distortions due to hand shake, creating the impression of a video shot on a tripod. For larger motions, we filter the camera's angular velocity to produce a smooth output. To meet the latency constraints of a real-time camera capture pipeline, our filter operates on a small temporal window of three to five frames. Our algorithm performs better than the previous work that uses a gyroscope to stabilize a video stream, and at a similar level with respect to current feature-based methods.

Specialized image signal processors (ISPs) exploit the structure of image processing pipelines to minimize memory bandwidth using the architectural pattern of line-buffering, where all intermediate data between each stage is stored in small on-chip buffers. This provides high energy efficiency, allowing long pipelines with tera-op/sec. image processing in battery-powered devices, but traditionally requires painstaking manual design in hardware. Based on this pattern, we present Darkroom, a language and compiler for image processing. The semantics of the Darkroom language allow it to compile programs directly into line-buffered pipelines, with all intermediate values in local line-buffer storage, eliminating unnecessary communication with off-chip DRAM. We formulate the problem of optimally scheduling line-buffered pipelines to minimize buffering as an integer linear program. Finally, given an optimally scheduled pipeline, Darkroom synthesizes hardware descriptions for ASIC or FPGA, or fast CPU code. We evaluate Darkroom implementations of a range of applications, including a camera pipeline, low-level feature detection algorithms, and deblurring. For many applications, we demonstrate gigapixel/sec. performance in under 0.5mm(2) of ASIC silicon at 250 mW (simulated on a 45nm foundry process), real-time 1080p/60 video processing using a fraction of the resources of a modern FPGA, and tens of megapixels/sec. of throughput on a quad-core x86 processor.

We present a solution for tagging features in OpenStreetMap using text recognition with an Android smartphone. The application extracts identifying text from a natural image taken with the phone and uses its GPS to tag the appropriate object on the map. Text detection is performed on the captured image by extracting maximally stable extremal regions, filtering and grouping blobs, and drawing a bounding box around the resulting designated ‘focus’ region. Adaptive binarization is then performed using K-means color selection, HSV thresholding, and connected component filtering. The resulting binary mask is processed using the Tesseract OCR engine, and the text is then displayed for confirmation and subsequently tagged on OpenStreetMap.