MapReduce Resource Allocation via Stage Assessment Model
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In MapReduce-based distributed systems, the allocation of computing resources to different stages of tasks is often done upfront, leading to inefficiencies due to varying complexities of algorithms across stages, resulting in wasted resources and suboptimal utilization.
Innovation Solution
A resource allocation method where a management node establishes a resource assessment model based on task data sizes and sends this information to a target computing node, which adjusts computing resources in real time for each stage according to actual requirements, using a preset resource pool and priority algorithms to optimize resource allocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If computing resources are allocated upfront to entire reduce tasks, then resource allocation simplicity is maintained, but computing resource utilization deteriorates due to waste from varying stage complexities
Solution Approach 1:
The patent segments the reduce task into multiple computing stages (first computing stage, second computing stage, etc.), each with potentially different complexity requirements. By dividing the task execution into discrete stages, the system can allocate computing resources dynamically at each stage boundary based on actual complexity needs, rather than allocating a fixed amount for the entire task. This segmentation enables precise resource matching while maintaining manageable allocation complexity through automated stage-based control.
2Loss of energy
If computing resources are dynamically adjusted for each computing stage, then computing resource utilization is improved, but system complexity increases
Solution Approach 1:
The patent performs preliminary actions by pre-defining multiple computing stages with different complexity levels before task execution. The resource assessment model is pre-configured with complexity parameters for each stage type. When a reduce task is received, the system automatically matches the task to appropriate stages based on pre-established complexity criteria, enabling dynamic resource adjustment without real-time complex calculations. This preliminary structuring reduces runtime complexity while maintaining dynamic adaptability.
Solution Approach 2:
The patent implements feedback mechanisms where the resource assessment model continuously evaluates task complexity at each stage transition point. Based on feedback from previous stage execution and task characteristics, the system adjusts computing resource allocation for subsequent stages. This feedback loop enables automated dynamic resource adjustment without requiring complex manual intervention, as the system self-regulates resource distribution based on observed task requirements.
3Productivity
If a fixed resource model is used for all reduce tasks, then allocation speed is maintained, but adaptability to different task complexities deteriorates
Solution Approach 1:
The patent transforms the static resource allocation model into a dynamic one by introducing multiple computing stages with variable complexity parameters. The resource assessment model adapts its resource allocation based on the specific characteristics of each stage and task. Rather than using a fixed resource model for all tasks, the system dynamically selects and adjusts resource allocation parameters according to the measured complexity of each computing stage, enabling both fast allocation and high adaptability through automated stage-based parameter adjustment.
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
Embodiments of the present invention disclose a computing resource allocation method, a related device, and a MapReduce-based distributed system. The system includes a management node and a target computing node. The management node is configured to: obtain M computing tasks, and establish a resource assessment model according to a data size of M computing tasks; and send some computing tasks of the M computing tasks and information about the resource assessment model to the target computing node. The target computing node is configured to receive the some computing tasks and the information about the resource assessment model that are sent by the management node, and before executing each computing stage of a target task, substitute input data of the computing stage into the resource assessment model to compute a resource size of a computing resource required for the computing stage; and compute the input data by using a computing resource that is of the resource size and that is in a preset resource pool. According to the embodiments of the present invention, computing resource utilization can be improved.