Tensor Processing Architecture With Networked MPUs and 3D Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning hardware struggles to efficiently perform complex computations required for deep learning applications, particularly in large neural networks, due to limitations in processing tensor operands and scalability.
Innovation Solution
A specialized ASIC-based deep learning hardware (DLH) device is designed to accelerate computations for deep learning applications by supporting batch-based and online training of networks, utilizing a network of interconnected matrix processing units that perform arithmetic and convolutional operations on tensor operands, with flexible architecture and high-bandwidth memory to support various machine learning network topologies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional machine learning hardware is used, then device complexity is reduced, but processing efficiency and computational performance deteriorate
Solution Approach 1:
The hardware is divided into multiple specialized processing units including matrix processing units (MPUs), convolution processing units (CPUs), and vector processing units (VPUs), each optimized for specific computational tasks. This segmentation enables parallel processing of different operations simultaneously, dramatically improving processing efficiency while maintaining manageable complexity through functional specialization.
Solution Approach 2:
The patent implements a three-dimensional memory architecture with stacked memory layers and through-silicon vias (TSVs) to enable vertical data flow. This dimensional change allows simultaneous access to multiple memory layers, increasing memory bandwidth and reducing data transfer bottlenecks without proportionally increasing device footprint or complexity.
2Productivity
If general-purpose hardware is used, then adaptability is improved, but computational performance for specific deep learning operations deteriorates
Solution Approach 1:
The processing units are designed with configurable parameters and programmable logic that allow them to perform multiple types of operations. The matrix processing units can be configured for different matrix dimensions and operation types, while convolution units can adapt to various kernel sizes and strides, providing both high performance and flexibility for different deep learning workloads.
Solution Approach 2:
The hardware architecture incorporates dynamically reconfigurable components that can adjust their operational characteristics based on the specific computational task. Control units can dynamically allocate resources between different processing units, and memory access patterns can be dynamically optimized, allowing the system to adapt to varying computational requirements while maintaining peak performance.
3Productivity
If traditional memory architecture is used, then device complexity is reduced, but memory bandwidth requirements and data transfer time increase
Solution Approach 1:
The patent implements a three-dimensional memory architecture with stacked memory layers and through-silicon vias (TSVs) to enable vertical data flow. This dimensional change allows simultaneous access to multiple memory layers, increasing memory bandwidth and reducing data transfer bottlenecks without proportionally increasing device footprint or complexity.
Solution Approach 2:
Buffer memory units are positioned as intermediary components between the processing units and main memory storage. These buffers temporarily hold data during processing operations, reducing the frequency and volume of data transfers to and from main memory, thereby effectively increasing bandwidth utilization and reducing transfer time.
Data Source
AI summary
A network of matrix processing units (MPUs) is provided on a device, where each MPU is connected to at least one other MPU in the network, and each MPU is to perform matrix multiplication operations. Computer memory stores tensor data and a master control central processing unit (MCC) is provided on the device to receive an instruction from a host device, where the instruction includes one or more tensor operands based on the tensor data. The MCC invokes a set of operations on one or more of the MPUs based on the instruction, where the set of operations includes operations on the tensor operands. A result is generated from the set of operations, the result embodied as a tensor value.