Privacy-Preserving Machine Learning via Knowledge Distillation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In business-to-business machine learning projects, data privacy concerns arise when multiple parties are involved, as participants may not trust sharing their private data, hindering the training of machine learning models, especially for hospitals that lack computation resources.
Innovation Solution
A private and interpretable machine learning framework that uses knowledge distillation with perturbation to protect data privacy by transferring perturbed knowledge from teacher modules to student modules, reducing the risk of data exposure through batch-by-batch queries and differential privacy techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If private data is shared between multiple parties to train machine learning models, then model training capability is improved, but data privacy security deteriorates
Solution Approach 1:
The patent introduces a trusted third party (the server) as an intermediary that coordinates the machine learning training process without accessing the raw private data. The server aggregates gradients from multiple parties and performs federated averaging to update model parameters, enabling collaborative model training while keeping sensitive data localized at each party's premises and preventing direct data exposure between participants.
Solution Approach 2:
The patent segments the data and computation across multiple independent parties, where each party maintains its own local data and computes local model updates independently. Only the aggregated gradient information is shared with the server, not the raw data itself. This segmentation allows model training to proceed while preserving data privacy by ensuring that no single party or intermediary can access or expose the full private datasets.
2Measurement precision
If data is exposed to enable model training, then prediction accuracy is improved, but trust between parties deteriorates
Solution Approach 1:
The server acts as a neutral intermediary that facilitates model training without compromising party trust. It collects gradients from all parties, performs secure aggregation, and distributes updated parameters back to parties. This intermediary role enables accurate prediction model training while maintaining trust, as each party contributes to the collective model without exposing its private data to others, and the server operates transparently without accessing raw data.
Solution Approach 2:
Instead of sharing raw data, the system shares copies of gradient information that encode model update directions without containing actual data. The gradients are computed locally from private data but only the aggregated gradient copies are transmitted to the server. This copying approach allows the model to learn from diverse data sources and improve prediction accuracy while maintaining trust, as the shared information is transformed into abstract update instructions rather than exposed data.
3Productivity
If computation resources are increased to train models, then model performance is improved, but cost deteriorates
Solution Approach 1:
The patent merges the computation resources of multiple parties through federated averaging. Each party performs local training computations using its own data and computation capabilities, then the server aggregates these local results to produce a global model update. This merging approach enables high model performance by leveraging the combined computational power and data diversity of all parties, while avoiding the need for any single party to deploy expensive large-scale models independently, thus reducing overall system cost.
Data Source
AI summary
Approaches for private and interpretable machine learning systems include a system for processing a query. The system includes one or more teacher modules for receiving a query and generating a respective output, one or more privacy sanitization modules for privacy sanitizing the respective output of each of the one or more teacher modules, and a student module for receiving a query and the privacy sanitized respective output of each of the one or more teacher modules and generating a result. Each of the one or more teacher modules is trained using a respective private data set. The student module is trained using a public data set. In some embodiments, human understandable interpretations of an output from the student module is provided to a model user.