Distributed Generative Models for Bandwidth-Efficient ML Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The physical limitations of computer networks, such as bandwidth constraints, hinder the deployment of machine learning models across distributed locations, and the scarcity of fully labeled data for model training, especially in scenarios with partially labeled data from various sites.
Innovation Solution
A service receives machine learning-based generative models from distributed sites, which are trained locally to generate synthetic unlabeled data, and combines this with subsets of labeled data to train a global machine learning model, reducing the need for data aggregation and leveraging partially labeled data across multiple sites.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If training data is transmitted from all distributed locations to a central entity, then centralized machine learning model training can be achieved, but network bandwidth consumption increases prohibitively
Solution Approach 1:
The patent segments the centralized training process into distributed local training at each site and centralized aggregation of results. Each site trains a generative model locally using its own data without transmitting the data itself, thereby segmenting the training workload and avoiding bulk data transmission while still achieving comprehensive model training.
Solution Approach 2:
The patent uses generative models to create synthetic copies of unlabeled data at each distributed site. These synthetic data copies are then transmitted to the central entity instead of the original data, allowing the central model to learn from diverse data distributions without the bandwidth cost of transmitting actual sensitive or voluminous data sets.
2Productivity
If only partially labeled data is available from distributed sites, then data collection is easier and less labor intensive, but model training quality deteriorates due to lack of ground-truth labels
Solution Approach 1:
The patent applies preliminary unsupervised pre-training of generative models at each distributed site using the available unlabeled data. This preliminary action extracts useful patterns and representations from the unlabeled data before the centralized supervised training phase, thereby preparing the data in advance to maximize the value of limited labeled data during final model training.
Solution Approach 2:
The patent merges unsupervised learning from unlabeled data with supervised learning from labeled data in a two-stage process. First, generative models are trained unsupervised on local unlabeled data to capture data distributions. Second, these models generate synthetic labeled data that is combined with actual labeled data for centralized supervised training, thereby merging the benefits of both labeled and unlabeled data sources.
3Reliability
If all distributed data is aggregated centrally for training, then comprehensive model training is achieved, but communication costs and time increase significantly
Solution Approach 1:
The patent segments the training process so that computationally intensive generative model training occurs locally at each distributed site simultaneously, rather than requiring sequential data aggregation and centralized processing. This parallel segmentation of training tasks dramatically reduces the time required while maintaining comprehensive model training through centralized aggregation of synthetic data and model parameters.
Data Source
AI summary
In one embodiment, a service receives machine learning-based generative models from a plurality of distributed sites. Each generative model is trained locally at a site using unlabeled data observed at that site to generate synthetic unlabeled data that mimics the unlabeled data used to train the generative model. The service receives, from each of the distributed sites, a subset of labeled data observed at that site. The service uses the generative models to generate synthetic unlabeled data. The service trains a global machine learning-based model using the received subsets of labeled data received from the distributed sites and the synthetic unlabeled data generated by the generative models.