NB-SMT Precision Scaling for Non-Blocking DNN Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) face inefficiencies due to unstructured sparsity and computational burdens, leading to underutilization of hardware resources and energy inefficiencies, despite their predictable control flow and tolerance to reduced precision.
Innovation Solution
Non-blocking simultaneous multithreading (NB-SMT) leverages DNN algorithmic resiliency by temporarily reducing numerical precision to accommodate multiple threads concurrently, sharing hardware resources efficiently without stalling operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple threads are executed concurrently on shared hardware resources, then hardware utilization and productivity are improved, but structural hazards cause blocking and reduce speed
Solution Approach 1:
The patent changes the numerical precision parameter of DNN calculations dynamically. When thread collisions are detected on shared computational resources, the system reduces precision (e.g., from 32-bit to 16-bit or 8-bit) to allow multiple threads to proceed concurrently without blocking, thereby maintaining high hardware utilization while avoiding execution stalls.
Solution Approach 2:
The system dynamically adjusts calculation precision based on runtime conditions such as thread occupancy and resource availability. This dynamic parameter adjustment enables the system to switch between high-precision single-threaded execution and lower-precision multi-threaded execution, optimizing both speed and productivity adaptively.
2Productivity
If numerical precision is reduced to accommodate multiple threads, then productivity and energy efficiency are improved, but accuracy deteriorates
Solution Approach 1:
The patent applies precision reduction selectively and partially. Instead of reducing precision for all calculations, the system identifies specific DNN layers or operations where precision can be reduced without significant accuracy loss, applying lower precision only to those portions while maintaining full precision elsewhere.
Solution Approach 2:
The system incorporates feedback mechanisms to monitor the impact of precision reduction on output accuracy. Based on this feedback, the system can adjust the degree of precision reduction applied, ensuring that productivity gains do not come at the cost of unacceptable accuracy degradation.
3Use of energy by moving object
If unstructured sparsity is exploited to increase efficiency, then energy consumption is reduced, but hardware complexity increases
Solution Approach 1:
The patent designs computational units that can handle both dense and sparse data formats using the same hardware infrastructure. The system employs universal computation units that automatically detect and exploit sparsity patterns in the data, performing skip operations for zero-valued inputs without requiring dedicated sparse processing hardware.
Solution Approach 2:
The system enables data to self-indicate its sparsity characteristics through formatting conventions (e.g., compressed sparse row format). The hardware automatically detects these characteristics and adjusts its operation accordingly, eliminating the need for complex external control logic to manage sparsity exploitation.
Data Source
AI summary
A method for non-blocking multithreading, the method may include (a) providing, during a deep neural network (DNN) calculation iteration, to a shared computational resource, input information units related to multiple DNN threads; (b) determining whether to reduce a numerical precision of one or more DNN calculations related to at least one of the multiple DNN threads, and (c) executing, based on the determining, DNN calculations on at least some of the input information units to provide one or more results of the DNN processing.