Task Manager for Distributed Computing with Elastic Nodes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In computing systems with a mix of stable and elastic computing nodes, the uncertainty in the availability of elastic nodes leads to inefficient use of resources, as tasks are often interrupted when these nodes are recycled, resulting in idle stable nodes and increased costs for users seeking to execute high-throughput tasks.
Innovation Solution
A task manager is implemented at a stable computing node to dynamically request and allocate elastic computing nodes based on task demands, distribute tasks into sub-tasks for continued execution, and manage resource allocation to optimize task execution across both stable and elastic nodes, ensuring efficient use of resources and minimizing re-execution overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If users prefer to use computing nodes with longer available duration (stable nodes), then task execution reliability is improved, but computing resource utilization deteriorates because many elastic nodes are left idle
Solution Approach 1:
The patent segments tasks into multiple subtasks that can be executed independently on different computing nodes. This allows the system to distribute tasks across both stable and elastic nodes, utilizing elastic nodes for subtasks while maintaining overall task reliability through the segmentation approach.
Solution Approach 2:
The patent implements dynamic task distribution where the task manager adaptively assigns subtasks to available computing nodes based on current system state. This dynamic approach enables efficient utilization of elastic nodes when available while maintaining reliability by falling back to stable nodes when needed.
2Productivity
If elastic computing nodes are used to increase resource utilization, then computing resource utilization is improved, but task execution reliability deteriorates because tasks are interrupted when nodes are recycled
Solution Approach 1:
By dividing tasks into smaller subtasks, the system can execute them on elastic nodes with shorter available durations. Each subtask is independent and can be completed before node recycling, preventing task interruption while maintaining high resource utilization.
Solution Approach 2:
The task manager proactively monitors the available duration of elastic nodes and assigns subtasks in advance before node recycling occurs. This preliminary action ensures that subtasks are completed while the node is still available, preventing interruptions and maintaining reliability.
3Productivity
If tasks are divided into subtasks for execution on elastic nodes, then computing resource utilization is improved, but system complexity increases due to task management overhead
Solution Approach 1:
The task manager is designed as a universal component that handles multiple functions: task segmentation, subtask distribution, progress tracking, and result aggregation. This multi-functional design consolidates complexity into a single management entity rather than requiring separate mechanisms for each function.
Solution Approach 2:
The task manager acts as an intermediary between the user's original task and the distributed subtasks on multiple nodes. It absorbs the management complexity by providing a unified interface for task submission and result retrieval, shielding users from the underlying distribution complexity.
Data Source
AI summary
A method, apparatus, device and medium for executing tasks in a computing system are provided in the present disclosure. The computing system comprises a first computing node and a plurality of second computing nodes, available duration of the first computing node is known, and the available duration of the plurality of second computing nodes is unknown. A task manager is started at the first computing node, the task manager being used to manage a plurality of tasks to be executed in the computing system. The task manager requests a set of second computing nodes in the plurality of second computing nodes to the computing system. The task manager distributes a target task in the plurality of tasks to the set of second computing nodes so as to execute the target task by using the set of second computing nodes.