In-Memory Logical Engine for Deep Learning Kernel Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In Processing In-Memory (PIM) architectures, deep learning workloads face significant execution time bottlenecks due to the high computational requirements of convolution operations, particularly because private kernels for each input neuron result in low temporal locality and high bandwidth pressure, leading to inefficient processing and energy consumption.
Innovation Solution
The method involves identifying data portions from distinct data objects to be processed using the same logical operation, storing them in buffers for batch processing with a shared kernel, and utilizing a resistive memory array as a logical engine to perform matrix-vector multiplications, thereby reducing kernel retrieval frequency and achieving temporal locality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If private kernels are used for each input neuron in PIM architectures, then processing can be performed locally in memory, but temporal locality is low and bandwidth pressure is high
Solution Approach 1:
The patent merges multiple private kernels into shared kernels that can be reused across different input neurons. By combining kernels that were previously stored separately in memory, the system reduces the number of kernel retrieval operations and improves temporal locality, thereby reducing energy consumption while maintaining processing speed.
Solution Approach 2:
The patent makes kernels universal by designing them to serve multiple input neurons rather than being dedicated to a single neuron. These shared kernels can be reused across different computational tasks and data objects, reducing bandwidth pressure and improving energy efficiency while maintaining high productivity through local processing.
2Productivity
If kernels are stored in memory for each input neuron, then processing can be performed, but kernel retrieval frequency is high
Solution Approach 1:
The patent combines multiple kernel storage locations into shared kernel buffers, reducing the total number of kernel retrieval operations. By merging kernels that serve multiple input neurons, the system decreases kernel retrieval frequency and reduces the time lost to memory access operations.
Solution Approach 2:
The patent performs preliminary actions by pre-loading and caching shared kernels in buffer memory before they are needed for processing. This allows kernels to be readily available when required, reducing retrieval time and preventing processing bottlenecks while maintaining high throughput.
3Reliability
If data portions from distinct data objects are processed separately, then data integrity is maintained, but processing pipeline utilization is low
Solution Approach 1:
The patent merges the processing of data portions from distinct data objects by applying shared kernels across multiple data objects in sequence. This approach maintains data integrity through proper isolation of data portions while improving pipeline utilization by keeping the processing unit continuously occupied with reusable kernels.
Solution Approach 2:
The patent ensures continuity of useful action by designing the processing pipeline to continuously operate on multiple data objects using shared kernels. By eliminating idle periods between processing different data objects and maintaining a steady stream of computational tasks, the system improves pipeline utilization while preserving data integrity through structured processing.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach reduces the frequency of kernel retrieval, enhances processing pipeline utilization, and decreases energy consumption by leveraging temporal locality and the efficiency of resistive memory arrays for deep learning tasks, such as face detection and image recognition.
Implementation Method 1
utilizing a resistive memory array as a logical engine to perform matrix-vector multiplications
Data Source
AI summary
In an example, a method includes identifying, using at least one processor, data portions of a plurality of distinct data objects stored in at least one memory which are to be processed using the same logical operation. The method may further include identifying a representation of an operand stored in at least one memory, the operand being to provide the logical operation and providing a logical engine with the operand. The data portions may be stored in a plurality of input data buffers, wherein each of the input data buffers comprises a data portion of a different data object. The logical operation may be carried out on each of the data portions using the logical engine, and the outputs for each data portion may be stored in a plurality of output data buffers, wherein each of the outputs comprising data derived from a different data object.