Distributed Array Task Scheduling for Out-of-Core Matrix Operations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large data arrays exceeding physical memory space pose challenges for matrix operations, requiring efficient distribution of compute tasks among multiple worker nodes to minimize I/O communication and optimize storage and communication overheads.
Innovation Solution
A distributed architecture with a master node and worker nodes that schedules tasks, partitions data blocks, and dynamically optimizes storage formats to overlap computations with I/O communications, leveraging Presto-based distributed arrays and R programming language extensions for parallel execution and incremental processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data arrays are stored in physical memory for matrix operations, then computation speed is improved, but memory capacity is exceeded when processing large datasets
Solution Approach 1:
The patent divides the large data array into multiple data blocks and distributes them across multiple worker nodes in a cluster. Each worker node processes a subset of blocks independently, enabling parallel computation that exceeds the capacity of single-machine memory while maintaining computational efficiency.
Solution Approach 2:
The patent transitions from single-machine in-memory computation to distributed out-of-core computation across a cluster. By adding the dimension of networked worker nodes and implementing a scheduler coordinate system, the system handles datasets larger than any single machine's memory capacity.
2Adaptability or versatility
If data blocks are frequently transferred between memory and disk for out-of-core computations, then processing of large datasets is enabled, but I/O communication overhead increases
Solution Approach 1:
The scheduler pre-assigns data blocks to worker nodes before computation begins. Worker nodes load and cache frequently accessed blocks in local memory before they are needed for computation, reducing the frequency and volume of I/O operations during the actual computation phase.
Solution Approach 2:
The system maintains data blocks in worker node memory for the duration of their utility across multiple computation tasks. The scheduler tracks block usage and keeps blocks pinned in memory while they are needed, eliminating repeated I/O operations for the same blocks and enabling continuous computation without idle I/O waits.
3Productivity
If compute tasks are distributed among multiple worker nodes, then processing throughput is improved, but coordination and scheduling complexity increases
Solution Approach 1:
The scheduler implements a unified control mechanism that handles multiple functions: task generation, worker node selection, data block assignment, progress tracking, and result aggregation. This single coordinating component manages the entire distributed computation workflow, simplifying the system architecture despite the complexity of distributed processing.
Solution Approach 2:
The scheduler continuously monitors worker node progress and adjusts task assignment dynamically. Workers report completion status and the scheduler reassigns blocks from completed tasks to pending tasks, optimizing resource utilization and balancing the workload across the cluster in real-time.
Data Source
AI summary
A technique includes distributing a plurality of tasks among a plurality of worker nodes to perform a processing operation on an array. Each task is associated with a set of a least one data block of the array, and an order of the tasks is defined by an array-based programming language. Distribution of the tasks includes, for at least one of the worker nodes, selectively reordering the order defined by the array-based programming language to regulate an amount of data transferred to the worker node.