Distributed Parallelism Tuning Using Execution Time Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed computing systems face challenges in setting reasonable parallelism, which affects computational efficiency and stability, particularly in large-scale cluster deployments.
Innovation Solution
A method for determining parallelism in distributed operations by acquiring data features, using an AI model to predict execution time, and employing a Leap-forward Grid Search Algorithm to find optimal parallelism settings, minimizing overhead and ensuring accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If parallelism size is increased to improve computational efficiency, then processing speed increases, but disk and network overhead increases
Solution Approach 1:
The patent dynamically adjusts the parallelism parameter based on data features and execution time predictions. By changing the parallelism parameter adaptively rather than using a fixed value, the system optimizes the balance between computational efficiency and resource overhead, resolving the contradiction between improved productivity and reduced energy loss.
2Productivity
If reasonable parallelism is set to improve system performance, then computational efficiency improves, but system complexity increases
Solution Approach 1:
The system performs self-optimization by automatically determining the optimal parallelism parameter through analyzing data features and predicting execution times. This self-service mechanism eliminates the need for manual configuration and reduces operational complexity while maintaining high system performance.
Solution Approach 2:
The patent implements a feedback mechanism where the system evaluates data features, predicts execution times for different parallelism values, and uses this information to determine the optimal parallelism setting. This closed-loop feedback process automates the optimization, reducing system complexity while improving performance.
3Productivity
If parallelism is optimized to reduce overhead, then resource utilization improves, but determination complexity increases
Solution Approach 1:
The system performs preliminary analysis of data features before executing the distributed operation. By pre-evaluating the data characteristics and predicting execution times for different parallelism values in advance, the system simplifies the optimization process and reduces the complexity of real-time determination while improving resource utilization.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
Provided in the embodiments of the present application are a parallelism determination method in a distributed operation, and an electronic device and a storage medium. The method comprises: acquiring data features in operation data corresponding to a distributed operation; acquiring candidate parallelisms in a candidate parallelism list, and determining an execution time in a candidate reduction stage according to the candidate parallelisms and the data features; and according to the execution time in the candidate reduction stage, determining, from among the candidate parallelism list, a target parallelism corresponding to the distributed operation.