Job Decomposition for Distributed Computing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed computing frameworks like MapReduce, Spark, and Ray require significant upfront design and programming efforts, making it difficult to convert existing programs into distributed computing models, especially in the context of high worker mobility in the Internet industry where programs need frequent modifications.
Innovation Solution
A job decomposition processing method that analyzes a source program to determine function call graphs and feature information, recommends execution strategies, and inserts a distributed computing framework like MapReduce, Spark, or Ray, allowing for distributed execution without modifying the source program, thereby reducing human intervention and ensuring repeatability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If distributed computing frameworks (MapReduce, Spark, Ray) are used, then computing efficiency and parallel processing capability are improved, but program complexity and design difficulty increase significantly
Solution Approach 1:
The patent introduces a job decomposition processing system as an intermediary between the source program and distributed computing frameworks. This system automatically analyzes source programs, decomposes them into executable tasks, and submits them to distributed computing resources without requiring users to manually design distributed programs. The intermediary handles the complexity of framework integration, task scheduling, and resource management, allowing users to benefit from distributed computing efficiency while avoiding program complexity.
Solution Approach 2:
The job decomposition processing system performs self-service by automatically analyzing source programs, identifying parallelizable operations, decomposing them into distributed tasks, and managing execution on computing resources. The system autonomously handles program analysis, task generation, resource allocation, and result aggregation without requiring user intervention in the complex distributed programming process, thus improving productivity while minimizing the burden on users.
2Speed
If existing programs are converted to distributed computing models, then processing speed and scalability are improved, but modification difficulty and time consumption increase
Solution Approach 1:
The system performs preliminary action by pre-analyzing source programs to identify parallelizable operations and decomposition opportunities before actual execution. The job decomposition processing system maintains a library of analysis results and decomposition patterns that can be reused across different programs and execution contexts. This preliminary analysis enables rapid conversion of existing programs to distributed models without requiring time-consuming manual redesign, thus improving processing speed while minimizing modification time.
Solution Approach 2:
The patent employs copying by generating multiple identical or variant copies of computational tasks that can be executed in parallel across distributed resources. Instead of fundamentally redesigning existing programs, the system creates replicated task instances from the original program logic, distributing them across multiple computing nodes. This copying approach preserves the original program structure while enabling parallel execution, thus improving processing speed without significant modification effort.
3Reliability
If manual program modification for distributed computing is performed, then execution control and optimization are improved, but human resource cost and subjectivity increase
Solution Approach 1:
The job decomposition processing system performs self-service by automatically analyzing source programs, determining optimal decomposition strategies, generating executable tasks, and managing distributed execution. The system autonomously handles program analysis, task generation, resource allocation, and result aggregation without requiring user intervention in the complex distributed programming process, thus improving productivity while minimizing the burden on users.
Solution Approach 2:
The system implements feedback mechanisms by monitoring execution performance, resource utilization, and task completion status in real-time. Based on this feedback, the job decomposition processing system dynamically adjusts task distribution, resource allocation, and decomposition strategies to optimize execution control. This automated feedback loop ensures reliable execution while eliminating the need for continuous human monitoring and adjustment, thus reducing human resource costs while maintaining execution reliability.
Data Source
AI summary
A job decomposition processing method for distributed computing, which comprises: analyzing a source program to be run by program static analysis to determine a function call graph contained in the source program; determining feature information of functions contained in the source program by program dynamics analysis or/and a program intelligent decomposition algorithm, wherein the feature information of the functions is used to characterize relevant information when each function is being running; decomposing the source program based on the feature information of the functions, a function relationship and available resource information of a computing platform to form an execution recommendation for each function on the computing platform, i.e., which hardware resources are used for computing each function; finally inserting a modifier in the source program and starting computation on the computing platform.