Dynamic Polynomial Degree for Encrypted Neural Network Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network operations on fully homomorphic encrypted data face limitations due to the use of low-order polynomials, which restrict the depth of neural networks and hinder high-performance accuracy, especially when trying to approximate activation functions like ReLU.
Innovation Solution
The method involves generating approximation polynomials for each layer of a neural network based on layer importance, determining the degree of these polynomials to minimize error while satisfying computation time conditions, and using these polynomials to perform neural network operations on homomorphically encrypted data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high order polynomials are used to accurately approximate ReLU, then approximation accuracy is improved, but computation time increases significantly requiring excessive bootstrapping
Solution Approach 1:
The patent changes the parameter of polynomial degree from fixed high order to variable order across different layers. By adjusting the degree parameter based on layer importance and computational constraints, the system achieves accurate approximation where needed while reducing computation time in less critical layers, resolving the contradiction between approximation accuracy and computation time.
Solution Approach 2:
The patent applies different polynomial degrees to different layers based on their importance and input data characteristics. Rather than using a uniform high-order polynomial throughout the network, the system locally optimizes the polynomial degree for each layer, achieving high approximation accuracy in critical layers while using lower degrees in less critical layers to reduce overall computation time.
2Loss of time
If low order polynomials are used to reduce computation time, then computation time is reduced, but neural network depth is limited and performance accuracy deteriorates
Solution Approach 1:
The system dynamically changes the polynomial degree parameter based on layer characteristics and computational constraints. This allows the neural network to achieve greater depth by using low-order polynomials in early layers and higher-order polynomials in later layers, thereby improving overall performance accuracy while maintaining reduced computation time.
Solution Approach 2:
The patent segments the neural network into different layers with different polynomial degree requirements. By dividing the network into segments (layers) and applying appropriate polynomial degrees to each segment based on its specific needs, the system overcomes the limitation of uniform low-order polynomials that restricted network depth and accuracy.
3Device complexity
If uniform polynomial degree is applied to all layers, then device complexity is reduced, but approximation accuracy deteriorates due to varying layer importances
Solution Approach 1:
The patent implements local quality by assigning different polynomial degrees to different layers based on their specific importance and data characteristics. This local optimization improves approximation accuracy for each layer's specific requirements while maintaining relatively simple configuration through automated degree selection based on layer metrics.
Data Source
AI summary
An operation method of performing a neural network operation of fully homomorphic encrypted data is provided. The operation method includes: receiving data for performing the neural network operation and receiving a parameter for generating an approximation polynomial corresponding to the neural network operation; obtaining layer information corresponding to layers of a neural network model, the layer information based on the data; determining importances of the layers, respectively, wherein the determining of the importances is based on the parameter and the layer information; generating an approximation polynomial approximating the neural network operation for each of the layers, wherein the generating is based on the layer importance; and generating an operation result by performing the neural network operation based on the approximation polynomial, wherein the parameter includes a computation time condition that the neural network operation must satisfy.