AI Processor Cache with Compression and DMA
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
AI processors face challenges in efficiently storing and processing large amounts of feature and kernel data due to limitations in external memory speed, necessitating a reduction in the number of read and write operations.
Innovation Solution
A cache system is introduced that compresses and decompresses feature and kernel data using a dataflow controller, scratchpad memory, external DMA controller, compression/decompression device, and transfer state buffer to minimize data transfer between the AI processor and external memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in external memory, then storage capacity is sufficient, but access speed is limited
Solution Approach 1:
The system divides the storage hierarchy into multiple segments: cache memory (fast, small capacity) and external memory (slow, large capacity). The cache stores frequently accessed feature data and kernel data, while external memory provides bulk storage. This segmentation allows the system to balance between storage capacity and access speed by keeping critical data in the faster cache segment.
Solution Approach 2:
The cache memory is nested within the AI processor, creating a hierarchical storage structure where the fast cache is embedded inside the processor while the larger external memory resides outside. This nested arrangement allows the processor to access frequently needed data from the internal cache without repeatedly accessing the slower external memory, thus resolving the speed-capacity tradeoff.
2Reliability
If data is frequently read and written from external memory, then data availability is maintained, but processing efficiency decreases
Solution Approach 1:
The system performs preliminary actions by pre-loading feature data and kernel data into the cache memory before they are needed for processing. The cache controller monitors data access patterns and proactively transfers data from external memory to cache, ensuring that required data is already available in the fast cache when processing begins, thereby maintaining data availability while improving processing efficiency.
Solution Approach 2:
The cache memory acts as an intermediary between the AI processor and external memory. It buffers data transfers, allowing the processor to access data from the fast cache rather than directly from external memory. This intermediary layer maintains data availability for processing while reducing the frequency of slow external memory accesses, thus improving overall processing efficiency.
3Quantity of substance
If large amounts of feature data and kernel data are stored, then computation requirements are met, but cache memory size becomes insufficient
Solution Approach 1:
The system changes the parameter of data representation by applying compression algorithms to feature data and kernel data before storing them in the cache. By transforming the data into a compressed format, the system can store larger volumes of effective data within the fixed cache memory size, thereby meeting computation requirements without increasing physical cache capacity.
Solution Approach 2:
The system implements a data management strategy where less frequently accessed data is temporarily discarded from the cache to make room for more critical data, with the understanding that discarded data can be recovered from external memory when needed. This approach allows the limited cache memory to dynamically hold the most important data for current computations while maintaining the ability to retrieve other data when required.
Data Source
AI summary
Disclosed is a cache including a dataflow controller for transmitting first data to a first processor and receiving second data from the first processor, an external direct memory access (DMA) controller for receiving the first data from an external memory to transmit the first data to the dataflow controller and receiving the second data from the dataflow controller to transmit the second data to the external memory, a scratchpad memory for storing the first data or the second data transmitted between the dataflow controller and the external DMA controller, a compression/decompression device for compressing data to be transmitted from the scratchpad memory to the external memory and decompressing data transmitted from the external memory to the scratchpad memory, and a transfer state buffer for storing transfer state information associated with data transfer between the dataflow controller and the external DMA controller.