Super Configuration Task for Distributed Job Workflow
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The complexity of distributed computing systems leads to decreased operation efficiency due to increased resource allocation and release times as the number of computing nodes grows, making it challenging to manage multiple jobs effectively without accessing source code.
Innovation Solution
Divide tasks into configuration and computation tasks, combine configuration tasks into a super configuration task, merge jobs into a super job, and execute these using multiple computing nodes, allowing for optimized resource allocation and release within a distributed computing environment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If the number of computing nodes in a distributed computing system increases to improve computing power, then the data computing power and computing performance are improved, but the time for allocating and releasing resources increases, which decreases the operation efficiency of jobs
Solution Approach 1:
The patent merges multiple configuration tasks from different jobs into a single super configuration task. This consolidation reduces the total time spent on resource allocation and release operations. By combining configuration tasks that were previously executed separately for each job, the system performs resource management operations more efficiently, thereby reducing the time loss associated with increasing the number of computing nodes.
2Reliability
If multiple jobs are executed serially with separate resource allocation and release tasks, then each job can be completed independently, but the ratio of execution time for managing and scheduling tasks to the entire application increases, affecting operation efficiency
Solution Approach 1:
The patent combines configuration tasks from multiple independently executed jobs into a super configuration task that can be executed once for all jobs. This merging reduces the overhead of repeated resource allocation and release operations while preserving the independent execution capability of each job through proper task dependency management in the workflow.
Solution Approach 2:
The super configuration task serves multiple jobs simultaneously, performing universal resource allocation and release functions that benefit all jobs in the workflow. This multi-functional approach allows a single configuration task execution to support multiple independent job executions, thereby improving overall operation efficiency without compromising job independence.
3Ease of operation
If configuration tasks are executed separately for each job, then resource management can be performed individually for each application, but the total time spent on resource allocation and release increases as the number of computing nodes increases
Solution Approach 1:
The patent merges individual configuration tasks into a super configuration task that handles resource management for multiple jobs collectively. This approach maintains the ease of individual resource management through structured task dependencies while reducing the total time spent on resource allocation and release by eliminating redundant operations across multiple job executions.
Data Source
AI summary
Embodiments of the inventive subject matter include receiving, in a distributed computing environment, a plurality of files for execution. Embodiments further include identifying, by parsing the plurality of files, code segments contained in each of the plurality of files. Embodiments further include determining, based on a comparison of the code segments and definitions contained in a distributed computing basic function library, a first group of the code segments that include configuration tasks and a second group of the code segments that include computational tasks. Embodiments further include combining the first group of the code segments to form a super configuration task. Embodiments further include creating an executable code, wherein the executable code comprises the super configuration task and the second group of code segments. Embodiments further include allocating the executable code to one or more nodes. Embodiments further include executing the executable code on the one or more nodes.