Deep Learning Processor Tensor Engine Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deep learning infrastructure, particularly GPUs and FPGAs, are inefficient and costly for the inference phase, failing to achieve both performance and energy efficiency needed for the growing demand of deep learning applications.
Innovation Solution
A hardware-based programmable deep learning processor (DLP) with multiple tensor engines, matrix multiplier engines, convolutional network engines, and vector floating-point units is introduced, optimized for the inference phase to provide high performance and energy efficiency by distributing neural network operations and utilizing sparsity and data reuse.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If GPUs and FPGAs are used for deep learning inference, then computational performance can be achieved, but energy efficiency and capital cost are poor
Solution Approach 1:
The system segments the deep learning inference workload by separating training operations (performed on GPUs) from inference operations (performed on specialized inference servers). The inference server further segments processing into multiple tensor engines, each handling specific neural network layers or operations, allowing optimized resource allocation for energy efficiency while maintaining performance.
Solution Approach 2:
The patent changes the architectural parameters of the computing system by introducing dedicated inference servers with specialized hardware (tensor engines, matrix multiplier engines, convolutional network engines) that are optimized for inference workloads. This parameter change from general-purpose GPUs to specialized inference hardware achieves both high performance and energy efficiency simultaneously.
2Productivity
If top of the line computing resources are used, then performance can be maintained, but the gap between application demand and available capacity increases
Solution Approach 1:
The inference server is designed with universal tensor engines that can handle multiple types of neural network operations (matrix multiplication, convolution, activation functions) through a unified architecture. This multi-functionality allows the system to adapt to different deep learning applications and models without requiring specialized hardware for each case, thereby improving scalability while maintaining performance.
Solution Approach 2:
The system implements dynamic resource allocation where the inference server can dynamically adjust computing resources based on workload demands. Multiple tensor engines can be activated or deactivated, and processing capacity can be scaled up or down to match application requirements, providing both high performance when needed and efficient resource utilization.
3Adaptability or versatility
If general purpose CPUs are used, then versatility is maintained, but energy efficiency and performance for deep learning inference are insufficient
Solution Approach 1:
The patent substitutes the mechanical/general-purpose CPU computing system with a specialized inference server architecture that uses tensor engines, matrix multiplier engines, and convolutional network engines. This substitution replaces general-purpose instruction execution with dedicated hardware circuits optimized for neural network operations, achieving dramatically improved performance per watt while maintaining versatility through programmable tensor engines.
Data Source
AI summary
A hardware-based programmable deep learning processor (DLP) is proposed, wherein the DLP comprises with a plurality of accelerators dedicated for deep learning processing. Specifically, the DLP includes a plurality of tensor engines configured to perform operations for pattern recognition and classification based on a neural network. Each tensor engine includes one or more matrix multiplier (MatrixMul) engines each configured to perform a plurality of dense and/or sparse vector-matrix and matrix-matrix multiplication operations, one or more convolutional network (ConvNet) engines each configured to perform a plurality of efficient convolution operations on sparse or dense matrices, one or more vector floating point units (VectorFPUs) each configured to perform floating point vector operations, and a data engine configured to retrieve and store multi-dimensional data to both on-chip and external memories.