Distributed Resource Allocation via Task Duration Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed computing, accurately specifying resource requirements for complex workflows is challenging due to varying resource needs over time, making it difficult to reserve necessary resources and determine task completion within allocated time frames.
Innovation Solution
A method that obtains duration information from historical task executions, sorts tasks into groups based on execution time, and determines a time-varying resource allocation to ensure tasks complete within their allocated resources, using techniques like dynamic programming to minimize resource waste.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If resource requirements are specified for each phase of a distributed compute job, then resources can be reserved to ensure task completion, but the complexity of specifying resource requirements accurately and concisely increases due to workflow complexity
Solution Approach 1:
The patent segments tasks into different groups based on their duration characteristics (short-duration tasks vs. long-duration tasks). This segmentation allows for simplified resource allocation by treating different task groups differently, rather than requiring complex specifications for each individual task, thus resolving the contradiction between reliability and specification complexity.
Solution Approach 2:
The patent changes the parameter of resource allocation from static to time-varying. By determining time-varying resource allocations based on task duration groups, the system can guarantee task completion while using a relatively simple specification approach that adapts to different time periods, reducing the overall complexity of resource requirement specification.
2Device complexity
If a single resource requirement is determined for a phase, then resource allocation can be simplified, but it becomes difficult to accurately reflect varying resource needs over time
Solution Approach 1:
The patent implements dynamic resource allocation by determining time-varying resource allocations for different task groups. Instead of using a single static resource requirement, the system adjusts resource allocations dynamically based on the time-varying nature of task execution, thereby maintaining both simplicity and accuracy.
Solution Approach 2:
By segmenting tasks into duration-based groups and applying different resource allocation strategies to each group, the patent achieves accurate resource requirement representation without requiring a single complex specification. Each segment can be allocated resources appropriately for its specific needs.
3Reliability
If resources are allocated to ensure tasks finish within allocation, then task completion is guaranteed, but resource waste increases due to over-reservation
Solution Approach 1:
The patent applies partial action by allocating resources specifically to task groups that need them (short-duration tasks receive resources to complete within allocation, while long-duration tasks are handled differently). This avoids excessive resource allocation to all tasks, reducing waste while still guaranteeing completion for time-sensitive tasks.
Solution Approach 2:
Different resource allocation strategies are applied to different task groups based on their local characteristics (duration). Short-duration tasks receive guaranteed resources to complete within allocation, while long-duration tasks receive different treatment. This localized approach ensures reliability where needed without causing system-wide resource waste.
4Productivity
If historical run information is used to determine resource requirements, then resource allocation can be optimized, but it becomes difficult to decide on a single resource requirement when phases run with varying numbers of resources
Solution Approach 1:
The patent changes the approach from analyzing varying resource numbers to analyzing task duration characteristics. By focusing on duration as a stable parameter rather than varying resource counts, the system can efficiently use historical data to inform resource allocation decisions without dealing with the complexity of varying resource configurations across different runs.
Solution Approach 2:
The patent groups tasks by duration to create homogeneous categories, making historical analysis more manageable. Instead of dealing with heterogeneous varying resource numbers, the system creates homogeneous task groups with similar characteristics, simplifying the extraction of useful patterns from historical data while maintaining allocation efficiency.
Data Source
Figure 1A
Figure 1B
Figure 2
AI summary
Duration information indicative of an amount of time taken by each of one or more tasks of a distributed compute phase of a distributed compute job in a distributed compute cluster to execute is obtained. The one or more tasks are sorted into one or more groups based on the duration information and a resource requirement is determined for each of the one or more groups. A time-varying allocation of resources of the distributed compute cluster for the phase is determined based on the resource requirement for each of the one or more groups.