Tile-Based Computing Architecture for Cache Miss Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional high-performance computing libraries face challenges with large-scale matrix operations due to poor data locality, irregular data access, and limited cache utilization, leading to frequent cache misses and low computing efficiency.
Innovation Solution
A computing architecture that divides large-scale data into tiles for storage in off-chip memory, utilizing on-chip cache for partial tiles and dependence data, and employs a network for data recombination and scheduling to improve memory access locality and cache utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If traditional high-performance computing libraries (CUBLAS, MKL) are used for large-scale matrix operations, then computing power is maximized through SIMD parallelism, but cache utilization becomes extremely low and memory bandwidth is severely restricted
Solution Approach 1:
The patent segments large-scale matrix data into tile-based blocks that can be efficiently stored and processed. By dividing the matrix into smaller tiles, the system can load multiple tiles into on-chip cache simultaneously, improving cache utilization while maintaining parallel computing capabilities. The tile size is optimized to fit cache capacity, allowing better data reuse within the cache hierarchy.
Solution Approach 2:
The patent introduces a new dimension of data organization by using 3D tile-based storage and processing instead of traditional 2D matrix operations. This allows data to be arranged in a three-dimensional space (rows, columns, and depth/tiles), enabling more efficient cache utilization by loading entire tiles rather than individual rows or columns, thereby reducing memory bandwidth pressure.
2Quantity of substance
If data is stored in traditional matrix format for large-scale problems, then complete data storage is required in off-chip memory, but data transfer between on-chip and off-chip memory creates huge pressure on memory bandwidth
Solution Approach 1:
The patent segments large matrices into smaller tiles that can be independently stored and processed. This segmentation allows the system to work with subsets of data that fit within on-chip cache, reducing the need for frequent data transfer between off-chip and on-chip memory. The tile-based approach enables data to be processed in chunks, improving memory bandwidth efficiency.
Solution Approach 2:
The patent employs preliminary data recombination and tiling before computation. By organizing data into tiles in advance and pre-loading relevant tiles into on-chip cache, the system minimizes the need for frequent data transfer during computation. The global scheduler pre-fetches and recombines data tiles before they are needed for computation, reducing memory bandwidth pressure.
3Adaptability or versatility
If data access follows irregular patterns in equation set solving and matrix operations, then flexibility in handling different algorithms is maintained, but data locality deteriorates causing frequent cache misses
Solution Approach 1:
The patent segments data into tiles that maintain spatial locality while supporting irregular access patterns. Each tile is a self-contained block that can be independently accessed, allowing the system to handle irregular access patterns without sacrificing too much data locality. The tile structure enables efficient cache utilization even when access patterns are not sequential.
Solution Approach 2:
The patent introduces a data recombination network as an intermediary between off-chip memory and the computing units. This recombination network dynamically reorganizes and reorders data tiles based on access patterns, improving data locality by bringing frequently accessed tiles closer to the compute units. The recombination network acts as a buffer that can handle irregular access patterns while maintaining efficient cache utilization.
4Area of stationary object
If on-chip cache size is limited, then chip area and cost are controlled, but the cache cannot store all data required for large-scale computations
Solution Approach 1:
The patent segments large datasets into smaller tiles that can be stored in limited on-chip cache. By dividing the data into manageable tiles, the system can fit more data into the fixed cache size through efficient tiling and blocking strategies. The tile-based approach allows the cache to store multiple tiles simultaneously, increasing the effective data volume that can be held in on-chip memory.
Solution Approach 2:
The patent implements continuous data recombination and tile management to maximize cache utilization. The global scheduler continuously monitors cache usage and dynamically recombines and replaces tiles to maintain optimal cache occupancy. This continuous management ensures that the limited on-chip cache space is always filled with the most useful data tiles, maximizing the storable data volume within the fixed cache area.
Data Source
AI summary
Computing architecture comprises an off-chip memory, an on-chip cache unit, a prefetching unit, a global scheduler, a transmitting unit, a pre-recombination network, a post-recombination network, a main computing array, a write-back cache unit, a data dependence controller and an auxiliary computing array. The architecture reads data tiles into an on-chip cache in a prefetching mode, and performs computing according to the data tiles; in the computing process of the tiles, a tile exchange network is adopted to recombine a data structure, and a data dependence module is arranged to process a data dependence relationship possibly existing between different tiles. According to the computing architecture, the data utilization rate can be increased, the data processing flexibility is improved, and therefore Cache Miss is reduced, and the memory bandwidth pressure is reduced.