Distributed Array Reshaping via Block Length and Stride Parameters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed computing systems, efficiently reshaping large data arrays stored across multiple locales is challenging due to the need for frequent data transfers between different computer systems, which can be computationally expensive and bandwidth-intensive.
Innovation Solution
A method is introduced where a computer system identifies block length and stride parameters to determine which data blocks need to be transferred between locales, allowing for the reconstruction of a local, reshaped array by initiating data block transfers and receiving blocks from various locales, thereby optimizing data distribution and reducing computational overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is frequently transferred between different computer systems to perform operations on distributed arrays, then operations can be performed on large data arrays that exceed single system memory capacity, but data transfer becomes computationally expensive and bandwidth-intensive
Solution Approach 1:
The patent segments the data array into blocks and organizes them in a range-partitioned flattened representation distributed across multiple locales. Each locale holds a portion of the array, and the segmentation enables selective data transfer only for required blocks rather than moving entire arrays, reducing computational expense and bandwidth consumption while maintaining the ability to work with large data arrays exceeding single system memory capacity.
2Adaptability or versatility
If data blocks are transferred between locales to reshape arrays, then the array can be reshaped to match computational needs, but data transfer overhead increases
Solution Approach 1:
The patent performs preliminary actions by pre-computing the mapping between the original array layout and the desired reshaped layout. It determines in advance which data blocks need to be transferred and their destination locations before initiating the actual data transfer. This preliminary computation of transfer requirements optimizes the reshaping process by minimizing unnecessary data movements and reducing overall transfer time while maintaining flexible array reshaping capability.
3Productivity
If range-partitioned flattened representation is used to store distributed arrays, then data can be efficiently partitioned across multiple locales, but reshaping operations become complex
Solution Approach 1:
The patent introduces an intermediary representation layer - the range-partitioned flattened representation - that mediates between the physical distributed storage layout and the logical array reshaping operations. This intermediate form standardizes how data is partitioned across locales while providing a systematic method to translate between different array shapes. The intermediary structure simplifies reshaping operations by providing a uniform approach to mapping data blocks between different layouts, reducing operational complexity while maintaining efficient data partitioning.
Data Source
AI summary
Embodiments are directed to reshaping a partitioned data array. In an embodiment, a computer system identifies a block length parameter that describes the number of data blocks in the range-partitioned flattened representation of the array that appear consecutively in each locale. The computer system then identifies a stride parameter that describes the amount of separation between data blocks in the range-partitioned flattened representation of the array that appear consecutively in a plurality of locales. Based on the identified block length parameter and the stride parameter, the computer system determines which of the data blocks on the plurality of locales are to be sent to other locales to produce a local version of the reshaped array. The computer system then receives data blocks from the different locales in the distributed system and reconstructs the array based on the received blocks to create a local version of the reshaped array.