Distributed Computing Parameter Optimization via Multi-Objective Genetic Algorithm
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for optimizing parameter configurations in distributed computing jobs fail to consider memory resource usage efficiently, leading to waste and low optimization efficiency, making it impossible to perform job-level optimizations due to dynamic cluster loads and low efficiency.
Innovation Solution
A method and system that determine a key parameter configuration set based on job programs, establish a performance prediction model using a neural network, and employ a multi-objective genetic algorithm to optimize execution time and memory resource usage, allowing for job-level configuration optimization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If existing parameter optimization tools are used, then execution time of job can be optimized, but memory resource usage is not considered leading to resource waste
Solution Approach 1:
The patent transforms the single-objective optimization problem (execution time only) into a multi-objective optimization problem by introducing memory resource usage as an additional optimization target. The system simultaneously optimizes multiple parameters including execution time and memory resource consumption, thereby resolving the contradiction between reducing execution time and avoiding memory waste.
Solution Approach 2:
The system incorporates feedback mechanisms by continuously monitoring both execution time and memory resource usage during job execution. This feedback is used to dynamically adjust parameter configurations, ensuring that optimization decisions consider both time efficiency and resource conservation, thus preventing memory waste while maintaining fast execution.
2Productivity
If frame-level optimization is performed, then some configuration improvement can be achieved, but job-level optimization is impossible due to low efficiency and dynamic cluster loads
Solution Approach 1:
The system performs preliminary analysis and identification of key parameters before actual job execution. By pre-processing and determining which parameters have the most significant impact on both execution time and memory usage, the system prepares optimization strategies in advance, enabling efficient job-level optimization even under dynamic cluster conditions.
Solution Approach 2:
The patent applies local quality by focusing optimization efforts on specific key parameters rather than uniformly optimizing all parameters. The system identifies and targets the most critical parameters that influence both execution time and memory consumption, thereby achieving efficient job-level optimization without the overhead of comprehensive parameter tuning.
3Manufacturing precision
If comprehensive parameter optimization is attempted, then optimization thoroughness can be improved, but optimization efficiency decreases due to large quantity of parameters and complex relationships
Solution Approach 1:
The system extracts and isolates the key parameters from the large set of configuration parameters. By identifying and separating the most influential parameters that affect execution time and memory usage, the system reduces the optimization problem to a manageable subset, maintaining optimization thoroughness while significantly improving efficiency.
Solution Approach 2:
The patent segments the parameter optimization problem into distinct components: execution time optimization and memory resource usage optimization. By dividing the complex multi-parameter optimization into separable sub-problems that can be addressed independently and then integrated, the system achieves comprehensive optimization without the computational burden of treating all parameters simultaneously.
Data Source
AI summary
The present disclosure relates to a method and system for optimizing a parameter configuration of a distributed computing job. The method includes: obtaining job programs of different distributed computing jobs, and determining a key parameter configuration set; obtaining a cluster status during execution of the distributed computing job, randomly generating a sample data set based on the key parameter configuration set and the cluster status, and establishing a performance prediction model; correcting the performance prediction model by using a multi-objective genetic algorithm and an optimization module configured with an optimal configuration selection strategy; obtaining a job program of a to-be-optimized distributed computing job and a cluster status during execution of the to-be-optimized distributed computing job, and determining a to-be-optimized key parameter configuration item combination; and inputting, to the performance prediction model, the to-be-optimized key parameter configuration item combination and the cluster status during execution of the to-be-optimized distributed computing job, and outputting a key parameter configuration item combination with a shortest execution time. The present disclosure can rapidly and effectively optimize the key parameter configuration.