Polynomial Approximation for Encrypted Deep Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deep learning models face challenges in processing encrypted data due to limitations in computations compatible with Homomorphic Encryption (HE), particularly with activation layers that lead to increased input ranges and errors, causing neural networks to 'explode' and preventing end-to-end encrypted data processing.
Innovation Solution
The method involves training neural networks with regularization loss to reduce input ranges of non-HE-friendly layers, replacing these layers with polynomial approximations, and using statistical analysis to determine suitable polynomial degrees for HE-compatible layers, resulting in a revised neural network that maintains performance while allowing encrypted data processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If non-polynomial layers (NPL) are used in deep neural networks, then the network can model complex non-linear relationships, but the input range expands causing the network to 'explode' and preventing end-to-end encrypted data processing
Solution Approach 1:
The patent transforms NPLs into polynomial layers by changing the functional form from non-polynomial (e.g., ReLU, sigmoid) to polynomial representations. This parameter transformation allows the network to maintain non-linear modeling capability while ensuring bounded input ranges that are compatible with homomorphic encryption operations, preventing the 'exploding' behavior
Solution Approach 2:
The patent replaces the mechanical computation of non-polynomial activation functions with polynomial approximations that are computationally compatible with encrypted data processing. This substitution enables the same non-linear transformations to be performed on encrypted inputs without requiring decryption, maintaining both functionality and security
2Reliability
If homomorphic encryption is applied to enable encrypted data processing, then data security is improved, but computation compatibility is limited particularly with activation layers
Solution Approach 1:
The patent changes the computational parameters of activation layers from non-polynomial operations to polynomial operations. Polynomial operations have mathematical properties that are compatible with homomorphic encryption schemes, allowing encrypted data to be processed through these layers without breaking the encryption, thus resolving the compatibility issue while maintaining security
3Reliability
If polynomial approximation is used to replace NPLs, then HE-compatible processing is enabled, but model accuracy may be compromised
Solution Approach 1:
The patent performs preliminary training of the polynomial layers using plaintext data before deploying the model for encrypted inference. This pre-training phase allows the polynomial approximations to learn optimal parameters that closely match the behavior of the original non-polynomial layers, ensuring high accuracy is achieved before the model is used in its encrypted processing mode
Solution Approach 2:
The patent creates polynomial copies of the original non-polynomial activation functions. These polynomial copies are designed to replicate the input-output behavior of the original functions as closely as possible within the constrained input ranges, maintaining model accuracy while enabling HE-compatible processing
Data Source
AI summary
An embodiment includes performing a dedicated training process on a non-polynomial neural network resulting in a trained polynomial neural network, where the training process includes performing a plurality of training iterations on the neural network, and performing, between the training iterations, loss processing that (i) minimizes a loss of the neural network and (ii) reduces a range of values to a non-polynomial layer (NPL) of the neural network. The embodiment estimates a range of input values to the NPL of the thus trained neural network. The embodiment forms a replacement layer for the NPL, where the replacement layer comprises a polynomial approximation of an operation performed by the NPL. The embodiment also generates a revised neural network by replacing the NPL of the trained neural network with the replacement layer.