DMA Memory Address Caching for Neural Network Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network (NN) applications face challenges with unpredictable memory access patterns, leading to cache thrashing and increased latency due to the limited capacity of on-chip caches, especially when processing large data structures, which existing DMA techniques fail to address effectively.
Innovation Solution
A direct memory access (DMA) controller caches memory addresses of large data structures by checking if external memory addresses are registered in the cache, copying data to internal memory, and updating cache lines with internal memory references, reducing the need for external memory access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in external memory and accessed directly by applications, then memory capacity is sufficient, but access latency increases and power consumption rises
Solution Approach 1:
The patent segments memory into multiple levels: external memory for bulk storage and an on-chip cache for frequent access. The DMA controller independently manages the cache, segmenting the memory management function from the main processor. This segmentation allows data to be stored in external memory while frequently accessed data is cached locally, reducing access latency without sacrificing capacity.
Solution Approach 2:
The on-chip cache acts as an intermediary between the application and external memory. The DMA controller populates this cache with data from external memory, and applications access data through the cache first. This intermediary layer provides fast access to frequently used data while maintaining the ability to store large quantities of data in external memory.
2Loss of time
If on-chip cache capacity is increased to reduce external memory access, then access latency decreases, but chip area and cost increase
Solution Approach 1:
Instead of providing a large cache for all possible data, the patent implements a smaller on-chip cache that stores only the most frequently accessed data. The DMA controller dynamically manages this limited cache space, evicting less frequently accessed data and bringing in new data as needed. This partial caching approach reduces chip area while still providing latency benefits for common access patterns.
3Productivity
If DMA techniques are used for data transfer, then data movement efficiency improves, but unpredictable access patterns cause cache thrashing
Solution Approach 1:
The DMA controller performs preliminary actions by prefetching data into the cache before applications need it. It monitors access patterns and proactively loads data that is likely to be accessed soon, reducing the frequency of cache misses. This preliminary action stabilizes cache performance by ensuring frequently accessed data is already in the cache when needed.
Solution Approach 2:
The system implements feedback mechanisms where the DMA controller monitors cache hit rates and access patterns. When cache thrashing is detected (high miss rates), the controller adjusts its prefetching behavior and eviction policies. This feedback loop stabilizes cache performance by adapting to changing access patterns and preventing sustained thrashing conditions.
Data Source
AI summary
A request to provide an application with direct memory access to data stored at an external memory address of an external memory is received. Responsive to determining that the external memory address is not registered in a cache, the data is copied from the external memory address to a first internal memory address within the internal memory. A first cache line, within the cache, associated with the external memory address is updated to include a reference to the first internal memory address. The data is provided from the internal memory to the application.