Circular Buffer for Tensor Computation Memory Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning accelerators struggle to perform efficiently in edge devices with strict power consumption constraints, particularly when running inference exercises using previously trained models in real-time applications such as virtual reality, augmented reality, and mixed reality.
Innovation Solution
The implementation of circular buffers as input and output buffers for tensor computations, allowing for efficient use of small local memory by wrapping addresses and utilizing a token counter to manage data chunk loading and computation suppression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the local memory size is increased to store entire layer input and output, then the memory capacity is improved, but the power consumption and device complexity increase
Solution Approach 1:
The patent divides the layer input and output data into multiple data chunks that are processed sequentially. Instead of loading the entire input and output into local memory at once, the system processes data in smaller segments, allowing the same limited memory to be reused for each chunk. This segmentation enables handling of larger computational workloads without proportionally increasing memory capacity or power consumption.
Solution Approach 2:
The patent implements dynamic address wrapping for the circular buffer, where the buffer index automatically wraps around when reaching the buffer boundary. This dynamic mechanism allows the same physical memory locations to be reused for different data chunks, effectively creating a virtual memory space larger than the physical memory capacity without additional hardware resources.
2Quantity of substance
If the local memory size is increased to accommodate larger input and output tensors, then the data storage capability is improved, but the device complexity and power consumption increase
Solution Approach 1:
The circular buffer structure serves multiple functions: it stores input data, stores output data, and enables address wrapping for both input and output operations. The same buffer memory is reused across different data chunks and computation iterations, making the memory system universal rather than requiring separate dedicated memory spaces for each function.
Solution Approach 2:
The patent introduces a time dimension to the memory usage by processing data chunks sequentially over time. Instead of requiring all data to be simultaneously available in memory (spatial dimension only), the system uses temporal sequencing to load, process, and unload data chunks in order, effectively expanding the usable memory capacity through time-based data management.
3Quantity of substance
If data chunks are loaded sequentially to the circular buffer, then the memory usage is optimized, but the computation speed may be reduced due to address wrapping operations
Solution Approach 1:
The patent performs preliminary actions by pre-calculating and storing the layer input and output sizes, determining the optimal chunk size and number of chunks before execution. The circular buffer is pre-configured with its size parameters, and the address wrapping logic is pre-established. This preliminary setup eliminates the need for complex runtime calculations, allowing the system to proceed with efficient sequential processing without speed penalties.
Data Source
AI summary
In one embodiment, a method includes receiving a token indicating a data chunk becomes available in a first circular buffer of a pre-determined size from a direct memory access component, determining that a computation is to be performed with data including the data chunk based on the token, and generating one or more addresses corresponding to one or more data chunks within the first circular buffer that are to be retrieved for the computation, where a generated address is subtracted by the pre-determined size of the first circular buffer when the generated address is greater than a pre-determined maximum associated with the first circular buffer, and where the generated address is added by the pre-determined size of the first circular buffer when the generated address is less than a pre-determined minimum associated with the first circular buffer.