Neural Network Training Homomorphic Encryption Dropout
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network training with homomorphic encryption is computationally intensive and limited to simple models due to high bandwidth requirements and noise accumulation, which hinders efficient regularization techniques like dropout, especially when dealing with complex datasets.
Innovation Solution
The method optimizes neural network training by using dropout techniques that reduce multiplication depth through neuron grouping and packing formations, allowing for efficient regularization without the need for masking, thereby improving performance and accuracy with homomorphically encrypted inputs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If homomorphic encryption is used to encrypt training data, then data security is improved, but computational intensity and bandwidth requirements increase
Solution Approach 1:
The patent segments the computational process by performing dropout operations at the ciphertext level rather than requiring full decryption. This segmentation allows selective computation on encrypted data, reducing overall computational intensity while maintaining security.
Solution Approach 2:
The patent changes the parameter of encryption application by using homomorphic encryption properties to perform specific operations (dropout, masking) directly on ciphertext without full decryption. This parameter change reduces computational intensity for specific operations while maintaining data security.
2Reliability
If homomorphic encryption is used for neural network training, then data privacy is protected, but noise accumulation limits model complexity
Solution Approach 1:
The patent extracts and addresses the noise accumulation problem by introducing specialized techniques for handling encrypted computations. By taking out the noise management as a separate concern and applying specific mitigation strategies, the patent enables more complex models while maintaining data privacy through homomorphic encryption.
3Reliability
If traditional dropout techniques are applied to homomorphically encrypted data, then regularization is achieved, but high bandwidth requirements hinder efficiency
Solution Approach 1:
The patent merges the dropout operation with the homomorphic encryption framework by performing dropout at the ciphertext level. This merging eliminates the need for separate decryption and re-encryption steps, reducing bandwidth requirements and improving training efficiency while maintaining regularization effectiveness.
Solution Approach 2:
The patent changes the operational parameter by applying dropout directly to encrypted data rather than decrypted data. This parameter change in the operational domain enables efficient regularization with reduced bandwidth requirements by leveraging homomorphic properties.
4Use of energy by moving object
If simple neural network models are used with homomorphic encryption, then computational requirements are met, but performance and accuracy are limited
Solution Approach 1:
The patent introduces dynamic techniques that adapt the computational approach based on the specific operations needed. By dynamically applying different homomorphic operations and optimization techniques, the patent enables more accurate models while keeping computational requirements manageable through efficient encrypted computation methods.
Data Source
AI summary
A method, a neural network, and a computer program product are provided that optimize training of neural networks using homomorphic encrypted elements and dropout algorithms for regularization. The method includes receiving, via an input to the neural network, a training dataset containing samples that are encrypted using homomorphic encryption. The method also includes determining a packing formation and selecting a dropout technique during training of the neural network based on the packing technique. The method further includes starting with a first packing formation from the training dataset, inputting the first packing formation in an iterative or recursive manner into the neural network using the selected dropout technique, with a next packing formation from the training dataset acting as an initial input that is applied to the neural network for a next iteration, until a stopping metric is produced by the neural network.