Managed Bulk Memory Tensor Layout Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Delays occur in processing tensors when elements are not arranged contiguously in memory, requiring additional instructions to re-arrange data, which affects transfer and processing efficiency in accelerator subsystems for machine intelligence applications.
Innovation Solution
A data processing system that determines the optimal arrangement of tensor elements based on operations to be performed and generates data transfer instructions for re-arranging data in external memory, ensuring elements are stored contiguously for efficient processing and transfer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If tensor elements are stored in arbitrary layout in external memory, then memory flexibility is maintained, but processing delays occur when elements are not arranged contiguously
Solution Approach 1:
The system performs preliminary analysis of the computational graph to determine the optimal layout for tensor elements before data transfer. The compiler generates layout transformation instructions in advance, re-arranging tensor elements in external memory to match the processing requirements of the accelerator subsystem, thereby eliminating processing delays during execution
Solution Approach 2:
The system dynamically changes the memory layout parameter of tensors based on the specific operations to be performed. By analyzing the computational graph and determining the optimal arrangement for reduction operations, the system transforms the memory layout to ensure contiguous storage of elements that need to be processed together, thus improving processing efficiency
2Productivity
If additional instructions are generated to re-arrange tensor elements, then processing efficiency is improved, but device complexity increases
Solution Approach 1:
The compiler acts as an intermediary between the host system and the accelerator subsystem. It analyzes the computational graph, determines optimal tensor layouts, and generates layout transformation instructions that are embedded in the program. This intermediary process automates the complexity management, allowing the system to achieve efficient data transfer without requiring complex manual intervention
Solution Approach 2:
The system performs self-optimization by automatically analyzing its own computational requirements and generating appropriate layout transformation instructions. The compiler examines the computational graph, identifies reduction operations, and autonomously determines the optimal tensor element arrangement, eliminating the need for external optimization tools or manual configuration
Data Source
AI summary
A system and method for providing a set of data transfer instructions for converting one or more tensors between two different layouts. A first layout is used for storage of the data in host memory. A second layout is used for storage of the data in external memory accessible to a subsystem. The subsystem acts as a work accelerator to the host, and reads the external memory and processes the data read from the external memory. The first layout may be a logical representation of the tensor. The second layout is optimised for transfer to and processing by the subsystem. The data transfer instructions for converting between the two layouts are generated in dependence upon an analysis of the instructions to be executed by the subsystem.