Virtualized Data Iterators for Neural Network Memory Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural networks and deep neural networks face performance issues due to inefficient data processing and memory management, leading to avoidable latency and computational overhead, particularly when iterating over large datasets.
Innovation Solution
The implementation of virtualized hardware iterators in a neural network environment, which allows for optimized data processing and memory management by enabling parallel processing, specifying data dimensions, and generating instructions for neuron processors, thereby improving overall performance and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a controller iterates over large amounts of data to apply processing operations in neural networks, then the processing operation can be applied, but latency increases and processing performance deteriorates
Solution Approach 1:
The patent segments the data iteration process into multiple independent hardware iterators that can operate in parallel. Each hardware iterator handles a specific portion of the data processing task, allowing simultaneous execution of multiple iteration operations across different data blocks, thereby reducing overall latency and improving throughput.
Solution Approach 2:
The patent introduces a hardware level of abstraction (another dimension) between the controller and the data processing operations. Hardware iterators are deployed as separate executable entities that can be instantiated multiple times and configured with different parameters, enabling parallel processing in a new dimensional space rather than sequential controller-based iteration.
2Productivity
If traditional memory management is used in neural networks, then memory operations can be performed, but additional computations are required and processing time increases
Solution Approach 1:
Hardware iterators are designed to autonomously manage their own iteration state, including tracking current data positions, managing loop counters, and controlling data flow without requiring controller intervention. This self-service capability eliminates the need for additional computational overhead in managing iteration state and reduces processing time.
Solution Approach 2:
The hardware iterator acts as an intermediary component between the controller and the data processing units. It assumes responsibility for managing iteration logic and data flow, freeing the controller from these tasks and reducing the computational burden on the overall system.
3Productivity
If multiple hardware iterators are deployed for parallel processing, then processing capacity increases, but system complexity increases
Solution Approach 1:
The hardware iterator is designed as a universal, configurable component that can be instantiated multiple times with different parameters to handle various data processing tasks. Each iterator instance is self-contained and can be independently configured, allowing parallel deployment without proportionally increasing system complexity. The same hardware iterator design serves multiple functions across different data blocks and processing stages.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The performance of a neural network (NN) and/or deep neural network (DNN) can limited by the number of operations being performed as well as management of data among the various memory components of the NN/DNN. Using virtualized hardware iterators, data for processing by the NN/DNN can be traversed and configured to optimize the number of operations as well as memory utilization to enhance the overall performance of a NN/DNN. Operatively, an iterator controller can generate instructions for execution by the NN/DNN representative of one more desired iterator operation types and to perform one or more iterator operations. Data can be iterated according to a selected iterator operation and communicated to one or more neuron processors of the NN/DD for processing and output to a destination memory. The iterator operations can be applied to various volumes of data (e.g., blobs) in parallel or multiple slices of the same volume.