Tensor Page-and-Stick Layout for Linear Memory Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems face challenges in efficiently handling and processing large, high-dimensional tensors due to their growing size, leading to issues with data access and computation, particularly in AI accelerators where chunking is not correlated to tensor dimensions and hardware limitations restrict control-related counter comparisons.
Innovation Solution
The method involves restructuring tensors into pages of sticks, optimizing the number of items per stick and page based on system resources, and employing a single loop for linear traversal, using machine instructions like LDMUPZ or LDMUP to load data with dummy values, and dynamic program flow to handle phase ends and address gaps.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If tensors are processed using conventional methods in AI accelerators, then computation can be performed, but data access efficiency deteriorates due to uncorrelated chunking and hardware limitations
Solution Approach 1:
The patent segments tensors into a hierarchical structure of chunks containing pages containing sticks, where each level is optimized for specific access patterns. This segmentation enables efficient data access by aligning with hardware capabilities while maintaining computation efficiency through structured organization of the segmented data.
2Adaptability or versatility
If chunking is not correlated to tensor dimensions, then hardware limitations are accommodated, but data access patterns become irregular and inefficient
Solution Approach 1:
The patent applies local quality by making chunking correlated to specific tensor dimensions rather than using uniform uncorrelated chunking. This creates regular access patterns for specific dimension traversals while maintaining adaptability to hardware limitations through the configurable hierarchical structure that can be tuned for different hardware capabilities.
3Device complexity
If control-related counter comparisons are restricted by hardware, then system complexity is reduced, but handling of phase ends and address gaps becomes more difficult
Solution Approach 1:
The patent uses preliminary action by pre-calculating and storing metadata about chunk boundaries, page boundaries, and stick offsets before processing begins. This preliminary organization of structural information enables the system to handle phase ends and address gaps without requiring complex runtime counter comparisons, thus reducing control logic complexity while maintaining full handling capability.
4Use of energy by moving object
If tensors are loaded page by page with full pages, then memory bandwidth is utilized efficiently, but partial pages cause address gaps and synchronization overhead
Solution Approach 1:
The patent applies partial action by loading only the necessary sticks from each page rather than entire pages, and by using dummy values to fill partial sticks. This approach maintains efficient memory bandwidth utilization by keeping data streams continuous while eliminating address gaps and reducing synchronization overhead by processing only relevant data portions.
Data Source
AI summary
The present disclosure relates to a method of accessing a n-dimensional tensor of elements in a memory by a computer system. The multidimensional tensor comprises two-dimensional arrays, herein referred to as pages, each page being configured to comprise a predefined number of one-dimensional arrays of elements, herein referred to as sticks. The method includes linearly loading page per page of the tensor, and doing the following for each page: loading the non-empty sticks of the page from the memory using a base address of the page and determining a base address for the subsequent page using the number of loaded sticks and using an address offset indicative of potential empty sticks of the page. In case the number of loaded pages reaches a chunk size, the chunk page counter may be reinitialized and the linear loading may be continued with a subsequent page.