Power SoftMax Self-Attention Layer for Homomorphic Encryption
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Self-attention mechanisms in homomorphic encryption deep learning architectures, particularly in transformer-based models, face challenges due to the non-polynomial nature of the SoftMax function, which is difficult to implement efficiently in environments that only support polynomial operations.
Innovation Solution
Replace the SoftMax function with a Power SoftMax function that uses powers instead of exponentiation before training, and approximate the division operation with polynomials after training, to create a HE-friendly self-attention layer that maintains numerical stability and reduces multiplication depth.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the traditional SoftMax function is used in self-attention mechanisms, then the model achieves good performance, but the computational complexity increases and noise levels increase in homomorphic encryption environments
Solution Approach 1:
The patent changes the mathematical parameters of the SoftMax function by replacing the exponential function with a polynomial approximation function. This parameter change transforms the original SoftMax computation into a form suitable for homomorphic encryption, reducing computational complexity and noise accumulation while maintaining the normalization property that ensures model performance.
2Reliability
If the traditional SoftMax function is used in self-attention mechanisms, then the model achieves good performance, but the noise levels in ciphertexts increase during homomorphic encryption processing
Solution Approach 1:
The patent changes the mathematical parameters of the SoftMax function by replacing the exponential function with a polynomial approximation function. This parameter change transforms the original SoftMax computation into a form suitable for homomorphic encryption, reducing computational complexity and noise accumulation while maintaining the normalization property that ensures model performance.
3Adaptability or versatility
If polynomial approximation is applied to the Power SoftMax function, then the HE-friendly self-attention layer is achieved, but the multiplication depth increases
Solution Approach 1:
The patent segments the polynomial approximation process by pre-computing and storing polynomial coefficients during the training phase. During inference, the system divides the polynomial evaluation into multiple smaller multiplication operations that can be executed in parallel or sequentially with reduced depth, making the computation more suitable for homomorphic encryption constraints.
4Productivity
If the self-attention layer is modified for HE processing, then the computational efficiency in HE environments improves, but the training process becomes more complex
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing polynomial coefficients during the training phase. This preprocessing step simplifies the inference process in homomorphic encryption environments, as the coefficients are ready for direct use without requiring complex real-time computations, thereby improving computational efficiency during HE processing.
Data Source
AI summary
Mechanisms are provided for optimizing a deep learning (DL) computer model for homomorphic encryption (HE) workload processing. The mechanisms receive an original DL computer model architecture that is to be optimized for HE workload processing, and modifying the original DL computer model architecture by replacing a self-attention layer of the original DL computer model with an HE friendly self-attention layer that comprises a Power SoftMax function that does not have exponent terms, to thereby generate a modified DL computer model architecture. The mechanisms execute a machine learning training of the modified DL computer model architecture, approximate one or more elements of the Power SoftMax function with polynomials to generate a trained HE optimized DL computer model, and output the trained HE optimized DL computer model for execution on HE workloads.