Differentially Private Machine Learning Model Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning algorithms fail to achieve accurate model training while protecting the privacy of individual user information, particularly in datasets containing sensitive data such as user inputs, passwords, and search histories.
Innovation Solution
A computing system and method for differentially private machine learning that selects a subset of client devices, provides a machine-learned model, receives local updates, determines a bounded-sensitivity data-weighted average of these updates, and generates an updated model with a user-level differential privacy guarantee, ensuring privacy without sacrificing model accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If machine learning models are trained using actual user data to improve accuracy, then model accuracy is improved, but user privacy is compromised
Solution Approach 1:
The patent introduces a differential privacy mechanism as an intermediary between the user data and the machine learning model. This mechanism adds carefully calibrated noise to the training process, allowing the model to learn from user data while preventing the extraction of sensitive individual information. The privacy budget (epsilon parameter) controls the trade-off between accuracy and privacy protection.
Solution Approach 2:
The patent modifies the training process by changing key parameters: adding noise with controlled variance, implementing subsampling of users, and using a privacy budget parameter (epsilon) to control the amount of privacy protection. These parameter changes enable the system to achieve both accuracy and privacy guarantees simultaneously.
2Object-affected harmful factors
If existing machine learning algorithms are used to protect user privacy, then user privacy is protected, but model accuracy deteriorates
Solution Approach 1:
The patent implements a dynamic training process where the amount of noise added and the subsampling rate are adjusted based on the privacy budget and training progress. The system dynamically balances between adding enough noise for privacy protection and maintaining sufficient signal for accurate learning, rather than using fixed aggressive privacy measures that would always degrade accuracy.
Solution Approach 2:
The patent applies partial differential privacy protection by subsampling only a portion of users in each training iteration and adding noise at controlled levels. This partial application of privacy mechanisms allows the model to learn from sufficient data while still providing privacy guarantees, avoiding the excessive privacy protection that would completely prevent accurate learning.
3Measurement precision
If user data is collected for model training, then model accuracy is improved, but data sensitivity increases
Solution Approach 1:
The differential privacy mechanism serves as an intermediary layer between the sensitive user data and the training process. It transforms the raw user data into privacy-preserving gradients or updates that can be used for training without exposing sensitive information. This intermediary processing ensures that even though user data is collected, the sensitive information cannot be recovered from the training outputs.
Data Source
AI summary
Systems and methods for learning differentially private machine-learned models are provided. A computing system can include one or more server computing devices comprising one or more processors and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors cause the one or more server computing devices to perform operations. The operations can include selecting a subset of client computing devices from a pool of available client computing devices; providing a machine-learned model to the selected client computing devices; receiving, from each selected client computing device, a local update for the machine-learned model; determining a differentially private aggregate of the local updates; and determining an updated machine-learned model based at least in part on the data-weighted average of the local updates.