Distributed Computing Task Splitting and Node Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed training methods face challenges such as improper data or model splitting, leading to inefficient computation, low communication efficiency between nodes, and poor aggregation of intermediate results.
Innovation Solution
A distributed computing method that optimizes task splitting and communication by configuring parallel modes, connection manners, and communication synchronization between computing nodes, using gradient or non-gradient optimization algorithms to enhance information synchronization efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data or model splitting is performed in distributed training, then computational tasks can be distributed across multiple nodes, but improper splitting leads to low communication efficiency between computing nodes
Solution Approach 1:
The patent applies segmentation by dividing the data computing task into multiple subtasks and further segmenting them according to different parallel modes (data parallel, model parallel, hybrid parallel). This allows flexible distribution of computational work across computing nodes while maintaining optimal communication patterns. The segmentation enables each node to process appropriately sized data or model portions, reducing unnecessary communication overhead.
Solution Approach 2:
The patent introduces multiple dimensions of parallelism (data parallelism, model parallelism, hybrid parallelism) to solve the communication efficiency problem. By adding these dimensional perspectives to task distribution, the system can optimize the balance between computation and communication across different axes, thereby reducing communication time while maintaining productivity.
2Productivity
If data or model splitting is performed in distributed training, then computational tasks can be distributed across multiple nodes, but improper splitting leads to poor aggregation effect of intermediate results
Solution Approach 1:
The patent implements feedback mechanisms through optimized information synchronization algorithms (gradient optimization or non-gradient optimization) that continuously monitor and adjust the aggregation process. Intermediate results from different computing nodes are aggregated with feedback control to ensure accuracy, allowing the system to maintain high productivity while achieving precise aggregation effects through iterative optimization.
3Power
If distributed training is implemented to train large models on large data sets, then computational capacity is increased, but hardware restrictions and storage overhead increase
Solution Approach 1:
The patent extracts and optimizes the storage requirements by separating computation-intensive operations from storage-intensive operations in the distributed training framework. Through efficient data loading strategies and optimized intermediate result aggregation, the system reduces storage overhead while maintaining high computational capacity across the distributed hardware infrastructure.
Data Source
AI summary
The present disclosure relates to the field of data processing. Provided is a distributed computing method, comprising: acquiring a data computing task; splitting the data computing task to obtain subtasks, deploying the subtasks to computing nodes, and configuring a parallel mode for each of the computing nodes in a distributed training universal frame; configuring a connection manner and a communication synchronization manner between the computing nodes; optimizing information synchronization efficiency for the computing nodes by using a gradient optimization algorithm or a non-gradient optimization algorithm; and aggregating intermediate results generated by the computing nodes, and outputting a corresponding final computing result. The present disclosure may reduce restriction from a hardware system, and by means of effective distributed algorithm design, a subtask training space is reduced, and the model training time is reduced, thereby effectively improving the accuracy of model training, and reducing the storage overhead of gradient and model parameter variables. Further provided are a distributed computing system, a distributed computing device, and a non-transitory computer-readable storage medium, which have the above beneficial effects.