Secure Softmax Parameter Updating Without Overflow
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In secure multi-party computing for machine learning, the use of a softmax function to normalize classification prediction values can result in data overflow due to very large numerical values exceeding the data representation range, leading to precision issues or unusable classification models.
Innovation Solution
Perform reduction processing on classification prediction values to bring them within a manageable data range, followed by normalization and updating the model parameters based on the normalized results, using methods like long division and approximate exponential operations to avoid overflow.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If softmax function is used to normalize classification prediction values, then normalization is achieved, but data overflow occurs due to very large numerical values exceeding data representation range
Solution Approach 1:
The patent applies preliminary action by performing reduction processing on classification prediction values before normalization. Specifically, it subtracts the maximum prediction value from all prediction values to shift them to a negative range, preventing overflow during subsequent exponential operations in the softmax function while maintaining normalization accuracy
Solution Approach 2:
The patent changes the parameter range of prediction values by transforming them from their original positive range to a negative range through subtraction of the maximum value. This parameter transformation ensures that exponential operations produce manageable values within the data representation range of secure multi-party computing, eliminating overflow while preserving the relative relationships needed for accurate normalization
2Adaptability or versatility
If exponential operation is performed on classification prediction values, then softmax normalization is enabled, but data representation range is exceeded causing overflow
Solution Approach 1:
The patent applies preliminary anti-action by pre-processing prediction values to counteract the harmful effect of overflow before it occurs. By subtracting the maximum value to create negative prediction values, the exponential operation produces values between 0 and 1 that fit within the data representation range, preventing overflow while maintaining softmax normalization functionality
3Measurement precision
If data overflow occurs in secure multi-party computing, then classification model precision declines or becomes unusable, but the model training process is disrupted
Solution Approach 1:
The patent prevents precision loss and model unusability by performing preliminary reduction processing on prediction values before they can cause overflow. This proactive approach ensures that all subsequent calculations remain within the valid data representation range of secure multi-party computing, maintaining both precision and model usability throughout the training process
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Disclosed are a model parameter adjusting method and device, a storage medium and a program product, which relate to the field of secure multi-party computing such as multi-party machine learning, federated learning, joint modeling or the like. The method includes: performing classification processing on input data by using a classification model obtained through training based on secure multi-party computing, to obtain classification prediction values of the classification model (S201); performing reduction processing on the classification prediction values (S202); performing normalization processing on the classification prediction values subjected to the reduction processing, to obtain a normalization result of the classification prediction values (S203); and updating a parameter of the classification model according to the normalization result of the classification prediction values (S204). Input values for a normalization method are reduced by means of performing the reduction processing on the classification prediction values, such that generation of data overflow can be avoided in a normalization process, thus the training precision of the classification model can be improved.