Random Fourier Features for Unbiased Softmax Sampling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The computational cost of training machine-learned models using softmax cross-entropy loss grows linearly with the number of classes, making it prohibitively expensive for large numbers of classes, and existing sampled softmax methods can provide biased estimates of the gradient if the sampling distribution does not accurately approximate the softmax distribution.
Innovation Solution
The use of Random Fourier Features to linearize a kernel that provides an unbiased estimation of the softmax distribution, allowing for low-bias negative sampling of classes, which reduces the computational cost of training and maintains accuracy by closely approximating the softmax distribution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If full softmax cross-entropy loss is used for training, then training accuracy is maintained, but computational cost grows linearly with the number of classes
Solution Approach 1:
The patent segments the full softmax computation by introducing a two-stage sampling approach: first sampling candidate classes based on a simplified distribution, then computing softmax only over these candidates. This divides the computational task from processing all N classes to processing a small subset k<<N, reducing complexity from O(N) to O(k).
Solution Approach 2:
The patent changes the temperature parameter in the softmax function to create a simplified sampling distribution that is easier to compute. By using a temperature-scaled version of the softmax distribution, the method enables efficient sampling without requiring full softmax computation, thus reducing computational cost while maintaining training quality.
2Productivity
If sampled softmax with uniform sampling is used, then computational cost is reduced, but gradient estimation becomes biased
Solution Approach 1:
The patent introduces an intermediary sampling distribution based on temperature-scaled softmax that bridges between uniform sampling and full softmax. This intermediate distribution provides better gradient estimates than uniform sampling while requiring less computation than full softmax, acting as a mediator that balances accuracy and efficiency.
Solution Approach 2:
The method uses feedback from the temperature-scaled distribution to guide the sampling process. By computing sampling probabilities based on the temperature-scaled softmax values, the algorithm feedback-adjusts which classes are sampled, ensuring that informative classes are selected while maintaining computational efficiency.
Data Source
AI summary
Systems and methods for low bias negative sampling of classes according to the sampled softmax method are described herein. The systems and methods can include training a machine-learned model for classifying inputs into one or more classes of a plurality of classes, each of the plurality of classes having an associated class embedding in a plurality of class embeddings. The systems and methods can include selecting, by the one or more computing devices, one or more negative classes from the plurality of classes based at least in part on a probability distribution approximating a softmax distribution, wherein the probability distribution is determined based at least in part on a Random Fourier Features map.