Deep Learning System Private Inference Hermite Polynomial Activation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning systems face challenges in protecting privacy-sensitive information during machine learning operations, particularly in scenarios where data needs to be processed without exposing personal data to service providers or disclosing trained models to clients, due to limitations in homomorphic encryption and multi-party computation, especially with non-linear operations like ReLU.
Innovation Solution
The implementation of a deep learning system using a convolutional neural network (CNN) with homomorphic encryption and multi-party computation, employing a polynomial activation function based on Hermitic expansion with Hermite polynomials to perform Fourier transforms, which reduces the computational cost and network communication requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If homomorphic encryption and multi-party computation are used for private inference, then privacy protection is improved, but execution time and computational cost increase
Solution Approach 1:
The patent pre-calculates and stores Hermite polynomial expansion coefficients and Fourier transform results during an offline phase. During online private inference, these pre-computed values are directly utilized, avoiding repeated complex calculations and significantly reducing execution time while maintaining privacy protection through homomorphic encryption
Solution Approach 2:
The patent replaces the traditional ReLU activation function with a polynomial approximation based on Hermite expansion. This polynomial copy can be efficiently computed using pre-calculated coefficients and Fourier transforms, providing a computationally lighter alternative that maintains functional similarity while reducing execution time in encrypted environments
2Device complexity
If traditional activation functions like ReLU are used in private inference, then implementation simplicity is maintained, but computational cost and network communication requirements increase
Solution Approach 1:
The patent substitutes the non-linear ReLU activation function with a polynomial-based activation function that can be efficiently computed through Fourier transforms. This substitution replaces a computationally expensive operation with a more efficient mathematical approach that leverages pre-calculated Hermite polynomial coefficients, reducing computational cost while maintaining implementation feasibility
Solution Approach 2:
The patent transforms the activation function from a direct non-linear operation (ReLU) to a polynomial expansion based on Hermite polynomials with predetermined coefficients. This parameter transformation allows the system to use efficient Fourier transform algorithms and pre-computed values, significantly reducing computational cost during private inference operations
Data Source
AI summary
A method of operating a deep learning system configured to perform private inferences, including performing a convolution operation with respect to input values; and outputting result values from the convolution operation using an activation function, wherein the activation function includes a Hermitic expansion using a Hermite polynomial as an eigenfunction to perform a Fourier transform.