Edge Server DLA Memory Layout for Low-Latency ANN Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in efficiently processing neural network computations with high energy consumption and computation time, particularly in edge servers, due to limitations in data access and processing capabilities of Central Processing Units (CPUs).
Innovation Solution
The integration of a Deep Learning Accelerator (DLA) with random access memory and separate memory access connections, allowing parallel processing and reduced data access bottlenecks, enables efficient computation of Artificial Neural Networks (ANNs) with minimal CPU intervention.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If edge servers use traditional CPU processing for ANNs, then general-purpose computing is maintained, but energy consumption and computation time increase significantly
Solution Approach 1:
The system segments processing tasks by separating CPU general-purpose operations from DLA specialized deep learning operations. The DLA handles matrix multiplications and neural network computations independently, while the CPU manages data preprocessing and postprocessing, reducing overall energy consumption and computation time.
Solution Approach 2:
The patent introduces an intermediary memory interface and data transfer mechanism between the DLA and CPU. This intermediary structure optimizes data flow, reducing memory access bottlenecks and enabling efficient parallel operations without increasing energy consumption.
2Speed
If edge servers process large datasets from cameras in real-time, then responsiveness is improved, but computation time and energy consumption increase
Solution Approach 1:
The system performs preliminary actions by pre-processing input data (such as image resizing, normalization, and feature extraction) on the CPU before transferring to the DLA. This preparation reduces the computational burden on the DLA during real-time inference, enabling faster processing with reduced computation time.
Solution Approach 2:
The patent implements continuous data flow from camera input through the DLA processing pipeline to output generation. The system maintains continuous operation by buffering incoming data and processing it in parallel streams, eliminating idle time and ensuring real-time responsiveness without increasing overall computation time.
3Loss of time
If edge servers perform deep learning inference locally, then latency is reduced, but device complexity increases
Solution Approach 1:
The DLA is designed as a universal accelerator that can handle multiple deep learning frameworks and neural network architectures (such as CNNs, RNNs, and Transformers) through a unified hardware interface. This multi-functionality reduces device complexity by providing a single versatile processing unit rather than specialized hardware for each algorithm type.
Solution Approach 2:
The system dynamically adjusts processing parameters such as precision (floating-point, integer, fixed-point), batch size, and activation functions based on the specific neural network model and input data characteristics. This parameter adaptability allows the same hardware to efficiently process different models without increasing physical complexity, maintaining low latency across diverse applications.
Data Source
AI summary
Systems, devices, and methods related to a Deep Learning Accelerator and memory are described. An edge server may be implemented using an integrated circuit device having: a Deep Learning Accelerator configured to execute instructions with matrix operands; random access memory configured to store first instructions of an Artificial Neural Network executable by the Deep Learning Accelerator and second instructions of a server application executable by a Central Processing Unit; and an interface to a communication device on a computer network. The Central Processing Unit may be part of the integrated circuit device, or be connected to the integrated circuit device. The server application may be configured to provide services over the computer network based on output of the Artificial Neural Network and input received from one or more local devices via a bus, or a wired or wireless local area network.