Distributed Array Partitioning for Data Transfer Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale data analysis applications requiring linear algebra operations often face inefficiencies due to extensive computing resource demands, particularly in distributed computing environments where data transfer between nodes is high and not optimally managed.
Innovation Solution
A distributed computing system is implemented with a master node and worker nodes that dynamically assign and co-locate array partitions across multiple computing devices to minimize data transfer, using a storage layer and task scheduling to optimize array computations and reduce data movement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is distributed across multiple computing devices for large-scale data analysis, then computing capability and processing speed are improved, but data transfer between nodes increases and system complexity increases
Solution Approach 1:
The patent segments large arrays into smaller array partitions that are distributed across multiple computing devices. Each computing device processes only the relevant partitions it holds, reducing the amount of data that needs to be transferred between nodes while maintaining parallel processing capabilities for improved speed.
Solution Approach 2:
The patent performs preliminary actions by pre-partitioning arrays and pre-assigning partitions to specific computing devices before execution. This preliminary organization of data and tasks reduces data transfer overhead during runtime by ensuring that relevant data is already in the appropriate locations for efficient processing.
2Productivity
If arrays are partitioned and distributed across computing devices, then data processing efficiency is improved, but data transfer between devices increases
Solution Approach 1:
The patent applies local quality by ensuring that array partitions are strategically assigned to computing devices based on local data access patterns and computational needs. This localized optimization minimizes the volume of data that must be transferred between devices while maximizing processing efficiency for each device's assigned partitions.
Solution Approach 2:
The patent uses copying mechanisms to replicate array partitions to multiple computing devices as needed. By copying partitions to devices that will process them, the system avoids repeated data transfers during different processing stages, thereby reducing total data transfer volume while maintaining high processing efficiency.
3Adaptability or versatility
If dynamic task assignment is implemented during run-time, then adaptability and resource utilization are improved, but system complexity and overhead increase
Solution Approach 1:
The patent implements dynamics by enabling the system to adaptively reassign array partitions to different computing devices during run-time based on changing computational needs, data access patterns, or device availability. This dynamic reassignment improves resource utilization and adaptability while managing complexity through automated decision-making frameworks.
Data Source
AI summary
A plurality of array partitions are defined for use by a set of tasks of the program run-time. The array partitions can be determined from one or more arrays that are utilized by the program at run-time. Each of the plurality of computing devices are assigned to perform one or more tasks in the set of tasks. By assigning each of the plurality of computing devices to perform one or more tasks, an objective to reduce data transfer amongst the plurality of computing devices can be implemented.