Machine Learning Compute Hardware for Zero-Squashing and Mixed Precision
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current graphics processing units (GPUs) face inefficiencies in handling machine learning operations due to sparse computations and sparsity-related overheads, leading to suboptimal performance in deep neural networks (DNNs) despite using dense compute methods.
Innovation Solution
Implement hardware optimizations such as zero squashing, low precision integer multiply, and unequal bit-length parallel processing to enhance GPU efficiency for machine learning tasks, including techniques to bypass zero operations and manage accumulator overflow, as well as support variable precision computations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If dense compute methods are used in GPUs for machine learning operations, then hardware resources are utilized efficiently, but performance deteriorates due to sparse computations and sparsity-related overheads
Solution Approach 1:
The patent extracts and removes zero operations from the computation pipeline through zero-squashing techniques. When a multiply operation involves a zero operand, the hardware automatically eliminates that operation entirely, preventing energy waste on unnecessary computations while maintaining accurate results for non-zero elements
Solution Approach 2:
The patent applies different processing qualities to different parts of the data. Instead of uniformly processing all elements, the system identifies and treats zero elements differently by eliminating them, while maintaining full precision for non-zero elements. This local differentiation optimizes both performance and energy efficiency
2Productivity
If zero-squashing techniques are implemented to bypass zero operations, then computational efficiency improves, but hardware complexity increases
Solution Approach 1:
The patent merges the zero-detection and operation-elimination functions into existing hardware components. The zero-squashing logic is integrated with the multiply-unit and instruction pipeline, combining multiple functions (zero detection, operation selection, result routing) into a unified hardware mechanism that improves throughput without proportionally increasing complexity
Solution Approach 2:
The zero-squashing hardware is designed to work with existing GPU architectures, making the optimization universally applicable to various machine learning workloads. The same hardware mechanism handles different scenarios (multiply-by-zero, add-zero, etc.) without requiring separate specialized circuits for each operation type
3Measurement precision
If variable precision computations are supported to accommodate different data types, then computational accuracy improves, but processing time increases
Solution Approach 1:
The patent implements dynamic precision adjustment where the system automatically selects appropriate precision levels based on data characteristics. The hardware dynamically switches between different precision modes (full precision for non-zero elements, reduced precision for zero elements) during the same computation, optimizing both accuracy and speed without manual intervention
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
One embodiment provides for a compute apparatus to perform machine learning operations, the compute apparatus comprising a fetch unit to fetch a single instruction having multiple input operands, wherein the multiple input operands have an unequal bit-length, a first input operand having a first bit-length and a second input operand having a second bit-length; a decode unit to decode the single instruction into a decoded instruction; an operand length unit to determine a smaller bit-length of the first bit-length and the second bit-length; and a compute unit to perform a matrix operation on the multiple input operands to generate an output value having a bit length of the smaller bit length.