Proxy Model Training for Privacy in Federated Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for cross-participant model training in machine learning face challenges in sharing model parameters while maintaining data privacy, as they can leak information about the underlying data composition and individual data instances, and require significant bandwidth and centralized systems, which are undesirable for sensitive data domains like healthcare.
Innovation Solution
Implementing a system where each participant uses a proxy model updated with private training data to share with others, using a differentially private algorithm to mask individual data effects and control privacy exposure, allowing local training and peer-to-peer model mixing without a central system, with a simpler proxy model architecture to reduce privacy costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If model parameters are shared between institutions, then model training can proceed without centralizing sensitive data, but information about underlying data composition and individual data instances may be leaked
Solution Approach 1:
The patent introduces gradient clipping as an intermediary mechanism that modifies the gradient updates before they are shared between institutions. By limiting the magnitude of gradients, the system prevents direct inference of individual data instances while still allowing model training to proceed. This intermediary process protects privacy without completely blocking information flow.
Solution Approach 2:
The patent transforms the model parameters by applying gradient clipping thresholds, changing the parameter values to remove sensitive information while preserving useful training signals. This parameter transformation allows the system to share updated models without exposing raw data or exact gradient directions that could reveal individual instances.
2Productivity
If federated learning with centralized consolidation is used, then model parameter updates can be aggregated, but significant bandwidth requirements and dependence on centralized system are imposed
Solution Approach 1:
The patent extracts the consolidation function from a centralized server and distributes it to each participating institution. Each institution independently averages the gradient updates received from others, eliminating the need for a centralized consolidation system while maintaining the collaborative training benefits.
Solution Approach 2:
Each institution performs its own model consolidation by independently averaging gradient updates from other participants. This self-service approach allows institutions to aggregate model improvements without relying on external centralized infrastructure, reducing bandwidth requirements and system complexity.
3Measurement precision
If complex deep models with large number of parameters are used, then model accuracy can be improved, but communication overhead and bandwidth requirements increase significantly
Solution Approach 1:
The patent extracts only the essential gradient update information needed for model improvement, discarding redundant parameter communications. By sharing only the necessary update signals rather than complete model states, the system maintains accuracy while reducing communication overhead.
Data Source
AI summary
A model training system protects data leakage of private data in a federated learning environment by training a private model in conjunction with a proxy model. The proxy model is trained with protections for the private data and may be shared with other participants. Proxy models from other participants are used to train the private model, enabling the private model to benefit from parameters based on other models’ private data without privacy leakage. The proxy model may be trained with a differentially private algorithm that quantifies a privacy cost for the proxy model, enabling a participant to measure the potential exposure of private data and drop out. Iterations may include training the proxy and private models and then mixing the proxy models with other participants. The mixing may include updating and applying a bias to account for the weights of other participants in the received proxy models.