Distributed Neural Network Training With Exclusive Parameter Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training machine learning systems requires extensive and costly data, often difficult to obtain due to privacy concerns, and existing methods struggle with data sharing and network traffic inefficiencies.
Innovation Solution
A distributed neural network training system where model parameters are shared among computer systems, with exclusive access and updates managed through a round-robin scheme or mutual-exclusion mechanisms, allowing parallel training on non-shared data subsets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If training data is shared among multiple computer systems to enable parallel training, then training efficiency and productivity improve, but data privacy and security deteriorate
Solution Approach 1:
The patent creates copies of model parameters and distributes them to multiple computer systems for parallel training. Each system works with identical initial parameters, allowing efficient parallel processing while the original training data remains localized and private to each system. The copying mechanism enables productivity improvement without compromising data privacy.
Solution Approach 2:
The patent introduces model parameters as an intermediary between training data and the learning process. Instead of sharing sensitive training data directly, systems exchange and update model parameters which serve as a mediator. This intermediary approach allows parallel training efficiency while maintaining data confidentiality, as the parameters contain transformed information rather than raw sensitive data.
2Productivity
If model parameters are updated by multiple computer systems simultaneously, then training speed improves, but parameter consistency and reliability deteriorate
Solution Approach 1:
The patent implements periodic updates where model parameters are updated in discrete steps or epochs rather than continuously. Multiple computer systems perform training operations during specific time periods, then synchronize their results periodically. This periodic action maintains parameter consistency while allowing high training speed during update intervals.
Solution Approach 2:
The patent incorporates feedback mechanisms where updated model parameters are communicated back to all computer systems in the distributed network. Each system receives feedback about parameter changes from others and adjusts its local copy accordingly. This feedback loop ensures parameter consistency across all systems while maintaining the ability to train in parallel, resolving the contradiction between speed and reliability.
Data Source
AI summary
Apparatuses, systems, and techniques to train a machine-learned model. In at least one embodiment, a plurality of training clients each obtain an exclusive right to update a model in turn, and each client trains said model with training data not accessible to other training clients.