ANN Data Caching Across Multi-Memory Tiers for Faster Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing solutions for processing artificial neural network (ANN) computations face challenges with CPUs and GPUs, including throughput limitations, power consumption, and programming complexity, while Field Programmable Gate Arrays (FPGAs) lack efficient integration with AI frameworks and fail to address massive data movement and computation scale requirements.
Innovation Solution
A system and method for caching data in ANN computations using multiple memories with varying access speeds and priorities, assigning frequently used data to faster memories and less frequently used data to slower memories, optimizing memory allocation and data movement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is repeatedly fetched from remote storage during neural network computations, then data availability is maintained, but computational efficiency and processing speed deteriorate due to repeated transmission delays
Solution Approach 1:
The patent applies preliminary action by pre-fetching data from remote storage and caching it in local memory before it is needed for computation. This allows the data to be readily available when the neural network processing requires it, eliminating transmission delays during the actual computation phase and thereby improving computational efficiency.
2Speed
If large volumes of data are cached in local memory, then data access speed improves, but memory resource consumption increases
Solution Approach 1:
The patent applies parameter changes by dynamically adjusting the cache size allocation based on the specific computational workload and data access patterns of the neural network. Rather than allocating fixed large memory capacity, the system adapts the caching parameters to match actual needs, improving data access speed while avoiding excessive memory resource consumption.
3Speed
If data is cached locally without selective criteria, then data access speed improves, but storage of unnecessary data increases memory usage
Solution Approach 1:
The patent applies local quality by implementing selective caching criteria that identify and cache only the specific data subsets that are actually needed for neural network computations. This targeted approach ensures that cached data is relevant and useful, improving access speed for critical data while avoiding the waste of memory resources on unnecessary data.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Systems and methods for caching data in artificial neural network computations are disclosed. An example method may comprise receiving, by a communication unit, data and a logical address of the data, the data being associated with the ANN, determining, by a processing unit coupled to the communication unit and to a plurality of physical memories and based on the logical address and physical parameters of the physical memories, a physical address of a physical memory of the plurality of physical memories, and performing, by the processing unit, an operation associated with the data and the physical address. The determination of the physical address can be based on a usage count of the data in the ANN computation or a time lapse between a time the data is written to the physical memory and a time the data is used in the ANN computation.