Versatile DNN Accelerator Architecture for Concurrent Multi-Model Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current DNN accelerators struggle to efficiently handle the concurrent execution of multiple DNN models due to varying computation and communication characteristics, resource contention, and inefficient off-chip memory access, limiting their performance and energy efficiency.
Innovation Solution
Versa-DNN employs a tile-based architecture with distributed buffers, a flexible Network-on-Chip (NoC), and a communication and computational aware scheduling algorithm to optimize off-chip memory access, reduce data duplication, and adapt to diverse computation and dataflow requirements, supporting multiple DNN models concurrently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a fixed architecture DNN accelerator is used, then it can be optimized for a specific DNN model, but it cannot efficiently handle multiple DNN models with varying computation and communication characteristics
Solution Approach 1:
The patent implements dynamic reconfiguration of the processing element array, allowing the accelerator to adapt its architecture at runtime. The processing elements can be dynamically partitioned into different arrays (e.g., first PE array, second PE array) with different dataflow configurations (e.g., weight-stationary, input-stationary, output-stationary, row-stationary) to match the specific requirements of different DNN models being executed concurrently.
Solution Approach 2:
The processing element array is segmented into multiple independent PE arrays that can be independently configured and allocated to different DNN models. This segmentation allows each PE array to be optimized for specific computation patterns while enabling concurrent execution of multiple DNN models with different characteristics.
2Adaptability or versatility
If multiple DNN models are executed concurrently on a shared accelerator, then resource utilization improves, but resource contention and inefficient memory access increase
Solution Approach 1:
The patent assigns different dataflow types to different PE arrays based on the specific requirements of each DNN model. Each PE array can use the most appropriate dataflow (weight-stationary, input-stationary, output-stationary, or row-stationary) for its assigned model, optimizing local computation efficiency and reducing unnecessary data movement and energy consumption.
Solution Approach 2:
The accelerator architecture provides universal support for multiple dataflow types within the same hardware platform. The system can dynamically switch between weight-stationary, input-stationary, output-stationary, and row-stationary dataflows across different PE arrays, enabling efficient concurrent execution of diverse DNN models with different computation and communication characteristics.
3Quantity of substance
If data is frequently accessed from off-chip memory, then larger datasets can be processed, but memory access latency and energy consumption increase
Solution Approach 1:
The patent implements on-chip buffers that pre-load and cache frequently accessed data (weights, inputs, outputs) before they are needed for computation. By anticipating data access patterns and pre-positioning data in on-chip memory, the system reduces latency and avoids frequent off-chip memory accesses during actual computation.
Solution Approach 2:
The on-chip buffers act as intermediaries between off-chip memory and the processing elements. These buffers absorb the bandwidth requirements of multiple concurrent DNN models, allowing PEs to access data from fast on-chip memory rather than slow off-chip memory, thereby reducing access latency and improving throughput.
Data Source
AI summary
Emerging applications utilize numerous Deep Neural Networks (DNNs) to address multiple tasks simultaneously. As these applications continue to expand, there is a growing need for off-chip memory access optimization and innovative architectures that can adapt to diverse computation, memory, and communication requirements of various DNN models. To address these challenges, Versa-DNN is a versatile DNN accelerator that can provide efficient computation, memory, and communication support for the simultaneous execution of multiple DNNs. Versa-DNN features three unique designs: a flexible off-chip memory access optimization strategy, adaptable communication fabrics, and a communication and computational aware scheduling algorithm. The off-chip memory optimization strategy improves performance and energy efficiency by increasing hardware utilization, eliminating excess data duplication, and reducing off-chip memory accesses. The adaptable communication fabrics include distributed buffers, processing elements, and a flexible Network-on-Chip (NoC), which can dynamically morph and fission to support distinct communication and computation needs for simultaneously running DNN models. Furthermore, Versa-DNN has a scheduling policy which manages the simultaneous execution of multiple DNN models with improved performance and energy efficiency.