Distributed Model Training with Gradient Threshold Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current model training methods reduce the training efficiency of models deployed on cloud servers due to inefficient handling of gradient data from user terminals.
Innovation Solution
A distributed training method where node devices filter gradient data based on a predetermined threshold to select target data that meets training conditions, sending only this target data to the server for model parameter adjustment, thereby improving training efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If all gradient data from user terminals is uploaded to cloud servers for model training, then the model can be trained using comprehensive data, but the training efficiency is reduced due to processing unnecessary data
Solution Approach 1:
The patent extracts and filters only the necessary gradient data from the complete gradient data obtained from user terminals. The filtering operation removes redundant or noisy gradient information while retaining the essential training signals, thereby improving training efficiency without losing critical model update information
Solution Approach 2:
The patent applies different quality standards to different portions of gradient data. By filtering gradient data based on specific criteria (such as gradient norm thresholds or importance weights), the system treats high-quality gradient information differently from low-quality information, optimizing the training process by focusing computational resources on the most valuable data
2Productivity
If gradient data is filtered based on a predetermined threshold, then training efficiency is improved by reducing data volume, but privacy computing resources may be compromised if important gradient information is discarded
Solution Approach 1:
The patent changes the parameter representation of gradient data through filtering operations. By adjusting the gradient data according to predetermined thresholds and criteria, the system transforms the raw gradient information into a refined form that maintains essential characteristics while removing unnecessary components, thus balancing efficiency and reliability
3Reliability
If noise addition processing is applied to gradient threshold and target data, then model training robustness is improved, but computational complexity increases
Solution Approach 1:
The patent applies noise addition processing selectively rather than uniformly to all gradient data. By adding noise only to specific portions of the gradient threshold and target data (such as through controlled probabilistic operations or localized noise injection), the system achieves robustness improvements while minimizing the increase in computational complexity
Data Source
AI summary
This specification discloses model training methods and apparatuses, storage media, and electronic devices. In embodiments of this specification, after obtaining a model parameter from a first server, a node device generates a target model based on the model parameter, trains the target model to obtain gradient data generated during the training of the target model, filters, based on a predetermined gradient threshold, out data, in the gradient data, that does not meet a training condition needed by the first server for model training, to obtain target data, and sends the target data to the first server. The first server adjusts the model parameter based on the target data and gradient data sent by another node device, generates a model, and deploys the generated model in the first server to train the generated model.