On-Chip Data Partitioning for ML Accelerator Bandwidth Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current hardware security designs for on-chip data processing, particularly in machine learning applications, face inefficiencies due to high on-chip storage costs and bandwidth bottlenecks, as they do not optimize data reading and writing operations, leading to excessive memory access requirements.
Innovation Solution
An on-chip data partitioning read-write system and method that partitions data between on-chip and off-chip storage mediums using a data partitioning strategy, performs pre-operational processing on address indices, and splices data using various strategies like ordinal, reversed, and random replacements, reducing memory access bandwidth through independent read-write ports and interconnection techniques like PCI, PCIE, and HT.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored entirely in on-chip storage medium, then data access speed is improved, but on-chip storage overhead and cost increase significantly
Solution Approach 1:
The patent segments data storage into two parts: hot data (frequently accessed) is stored in on-chip storage medium while cold data (less frequently accessed) is stored in off-chip storage medium. This segmentation allows the system to maintain fast access for critical data while reducing overall on-chip storage requirements, directly resolving the contradiction between access speed and storage overhead.
Solution Approach 2:
The patent applies local quality by optimizing storage location based on data characteristics. Frequently accessed data (weights, activations) are placed in on-chip storage for fast access, while less frequently accessed data are placed in off-chip storage. This localized optimization ensures that only the necessary portion of data resides in expensive on-chip memory, reducing storage overhead while maintaining access performance.
2Quantity of substance
If data is stored entirely in off-chip storage medium, then on-chip storage overhead is reduced, but bandwidth bottleneck limits performance
Solution Approach 1:
The patent implements preliminary action by pre-loading frequently accessed data (weights and activations) into on-chip storage medium before computation begins. This pre-positioning of data eliminates the need for repeated data transfers during computation, reducing bandwidth requirements and preventing performance bottlenecks while keeping on-chip storage usage optimized.
Solution Approach 2:
The patent ensures continuity of useful action by maintaining hot data in on-chip storage throughout the computation process. This allows computation units to continuously access required data without interruption or repeated off-chip transfers, sustaining high productivity while minimizing bandwidth consumption through efficient data retention.
3Adaptability or versatility
If conventional cache is used to balance bandwidth, then bandwidth disequilibrium is addressed, but data reading and writing cost increases
Solution Approach 1:
The patent extracts the cache function and replaces it with a data partitioning strategy that separates hot and cold data into different storage mediums. Instead of using a traditional cache hierarchy that incurs read/write overhead, the system directly stores frequently accessed data in on-chip memory and less frequently accessed data in off-chip memory, eliminating cache management costs while maintaining bandwidth balance.
4Productivity
If on-chip storage is increased to accommodate all data, then data reusability is optimized, but hardware cost and area increase
Solution Approach 1:
The patent applies dynamics by implementing a flexible data partitioning strategy that can dynamically adjust which data reside in on-chip versus off-chip storage based on access patterns. This dynamic approach allows the system to optimize data reusability efficiency by keeping frequently accessed data on-chip while minimizing on-chip storage area through intelligent data placement and eviction policies.
Data Source
AI summary
The present invention is directed to the storage technical field and discloses an on-chip data partitioning read-write method, the method comprises: a data partitioning step for storing on-chip data in different areas, and storing the on-chip data in an on-chip storage medium and an off-chip storage medium respectively, based on a data partitioning strategy; a pre-operation step for performing an operational processing of an on-chip address index of the on-chip storage data in advance when implementing data splicing; and a data splicing step, for splicing the on-chip storage data and the off-chip input data to obtain a representation of the original data based on a data splicing strategy. Also provided are a corresponding on-chip data partitioning read-write system and device. Thus, read and write of repeated data can be efficiently realized, reducing memory access bandwidth requirements while providing good flexibility, thus reducing on-chip storage overhead.