FHE-Friendly ML Model via Knowledge Distillation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing approaches for encrypted inference in machine learning face challenges in achieving accurate models within the multiplicative depth constraints of leveled homomorphic encryption schemes, often requiring ad-hoc simplifications or retraining with original data, which compromises accuracy or is impractical due to data unavailability.
Innovation Solution
A knowledge distillation framework is used to create a full homomorphic encryption (FHE)-friendly machine learning model by constructing a student model that mimics a pre-trained teacher model, using synthetic training data and FHE-compatible activation functions, allowing for privacy-preserving inference on encrypted data without losing accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a complex pre-trained model is used for accurate inference, then prediction accuracy is improved, but the model cannot be evaluated on encrypted data due to FHE computation depth constraints
Solution Approach 1:
The patent creates a student model that copies the predictive behavior of the complex teacher model. The student model is trained to mimic the teacher model's predictions on a validation set, effectively copying its decision-making patterns while using a simplified architecture compatible with FHE constraints.
Solution Approach 2:
The patent changes the architectural parameters of the model to make it FHE-compatible. This includes using a shallow neural network with polynomial activation functions instead of deep networks with non-polynomial activations, and controlling the multiplicative depth to fit within FHE constraints.
2Adaptability or versatility
If a shallow FHE-compatible model is constructed, then FHE constraint satisfaction is improved, but prediction accuracy deteriorates compared to complex models
Solution Approach 1:
The patent introduces a teacher model as an intermediary that bridges the gap between complex accurate models and simple FHE-compatible models. The teacher model generates soft labels that guide the training of the student model, enabling the student to achieve high accuracy despite its simplified architecture.
Solution Approach 2:
The patent performs preliminary training of a complex teacher model on the full dataset before distilling knowledge to the student model. This preliminary action allows the student model to learn from the teacher's pre-processed knowledge rather than learning from scratch, improving efficiency and accuracy.
3Adaptability or versatility
If ad-hoc simplifications are applied to a pre-trained model for FHE inference, then FHE constraint satisfaction is improved, but prediction accuracy is lost
Solution Approach 1:
The patent segments the modeling process into two distinct phases: teacher model training on the full dataset, and student model distillation on a validation set. This segmentation allows each model to be optimized for its specific purpose without compromising the other's performance.
Solution Approach 2:
The patent replaces the traditional direct training approach with a knowledge distillation mechanism. Instead of directly training a shallow model to achieve FHE compatibility (which loses accuracy), the system substitutes a two-stage distillation process that transfers knowledge from the complex teacher model to the simple student model.
4Measurement precision
If original training data is used to retrain a depth-constrained model, then prediction accuracy is improved, but data availability becomes problematic
Solution Approach 1:
The patent uses a validation set as a disposable resource for distillation. Instead of requiring access to the valuable original training data, the system uses a smaller validation set that can be publicly available or easily generated, sacrificing minimal information to achieve the distillation goal.
Data Source
AI summary
This disclosure provides a method, apparatus and computer program product to create a full homomorphic encryption (FHE)-friendly machine learning model. The approach herein leverages a knowledge distillation framework wherein the FHE-friendly (student) ML model closely mimics the predictions of a more complex (teacher) model, wherein the teacher model is one that, relative to the student model, is more complex and that is pre-trained on large datasets. In the approach herein, the distillation framework uses the more complex teacher model to facilitate training of the FHE-friendly model, but using synthetically-generated training data in lieu of the original datasets used to train the teacher.