Super-tiling CNN Processing Reduces External Memory Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Convolutional neural networks (CNNs) face challenges in efficiently processing complex tasks on devices with limited compute and memory resources, as they often require large memory access, which can bottleneck processing times due to slower external memory access compared to on-chip memory.
Innovation Solution
The method involves processing a portion of a tensor, known as a super tile, across multiple layers of a CNN, storing and overwriting these portions in on-chip memory to reduce the need for external memory access, and using layer groups to optimize processing and memory bandwidth efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If all values of the first set are stored in on-chip memory for processing, then processing speed is improved, but on-chip memory capacity is insufficient to hold all values
Solution Approach 1:
The patent divides the first set of values into multiple portions (first portion, fourth portion, etc.) that can be processed sequentially in the on-chip memory. This segmentation allows the system to handle larger datasets than the memory can hold simultaneously, resolving the contradiction between memory capacity and processing speed by processing data in manageable chunks.
Solution Approach 2:
The patent pre-loads the first portion of values into on-chip memory before processing begins, and pre-allocates buffer spaces for intermediate and final results. This preliminary preparation ensures that when processing starts, all necessary data is readily available in fast memory, maximizing processing speed without requiring the entire dataset to fit in memory simultaneously.
2Quantity of substance
If intermediate results are stored in external memory, then memory capacity is sufficient, but memory access time increases due to slower external memory speed
Solution Approach 1:
The patent introduces on-chip memory as an intermediary between external memory and the processing units. The first portion of input values is loaded into on-chip memory, processed through multiple layers, and the intermediate results are stored back in on-chip memory rather than external memory. This intermediary approach allows fast access to frequently needed data while using external memory only for bulk storage, resolving the contradiction between capacity and access speed.
Solution Approach 2:
The patent implements a nested memory hierarchy where on-chip memory (faster, smaller) is nested within the broader external memory system (slower, larger). The system selectively places frequently accessed data (input portions and intermediate results) in the inner on-chip memory layer, while less frequently accessed data resides in the outer external memory layer, optimizing the balance between capacity and access time.
3Reliability
If the entire first set of values is processed at once, then processing completeness is achieved, but resource utilization is inefficient due to limited on-chip memory
Solution Approach 1:
The patent implements continuous processing by immediately overwriting the first portion of input values with intermediate results as they are generated, rather than waiting for complete processing or storing intermediate results separately. This continuous reuse of memory space ensures that the on-chip memory is constantly utilized for productive purposes, maintaining processing completeness while maximizing resource utilization.
Solution Approach 2:
The patent discards the original input values (first portion) after they have been processed to generate intermediate results, and recovers the memory space by overwriting it with the new intermediate results. This discarding and recovering approach ensures that memory resources are continuously reused rather than wasted on retaining data that has already served its purpose, achieving both completeness and efficiency.
Data Source
AI summary
Techniques including receiving a first set of values for processing by a machine learning (ML) network, storing a first portion of the first set of values in an on-chip memory, processing the first portion of the first set of values in a first layer of the ML network to generate a second portion of a second set of values, overwriting the stored first portion with the generated second portion, processing the second portion in a second layer of the ML network to generate a third portion of a third set of values, storing the third portion, repeating the steps of storing the first portion, processing the first portion, overwriting the stored first portion, processing the second portion, and storing the third portion for a fourth portion of the first set of values until all portions of the first set of values are processed to generate the third set of values.