Spherical Random Features for Polynomial Kernel Approximation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Scaling kernel methods, particularly for polynomial kernels, is challenging due to high dimensional mappings and assumptions in Gaussian kernels that do not hold true for polynomial kernels, leading to computational inefficiencies and inaccurate approximations.
Innovation Solution
The spherical random Fourier (SRF) method approximates polynomial kernels by generating nonlinear randomized feature maps using a sum of Gaussian kernels, optimizing coefficient and standard deviation values to minimize error, and projecting negative values to zero, allowing for efficient computation of Fourier transforms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If polynomial kernels are used for machine learning, then classification accuracy is improved, but computational complexity and memory requirements increase significantly
Solution Approach 1:
The patent creates explicit feature maps that copy the essential properties of polynomial kernel mappings without requiring actual high-dimensional polynomial expansions. By generating randomized feature maps that approximate polynomial kernels, the system achieves similar classification performance with linear complexity, effectively copying the beneficial properties of polynomial kernels while avoiding their computational burden.
Solution Approach 2:
The patent transforms the polynomial kernel problem into a different parameter space by using randomized feature maps with learnable parameters. Instead of computing polynomial expansions with degree d, the system uses feature maps with dimensionality controlled by separate parameters, allowing independent optimization of accuracy and computational efficiency through parameter tuning.
2Reliability
If high-order polynomial kernels are used, then model accuracy improves, but training and testing time increase
Solution Approach 1:
The patent copies the functional behavior of high-order polynomial kernels through randomized feature maps, achieving similar accuracy without performing actual high-order polynomial computations. The feature maps replicate the kernel's decision boundary properties while maintaining linear computational complexity during training and inference.
Solution Approach 2:
The patent segments the polynomial kernel computation into independent randomized feature generations. Instead of computing a single high-order polynomial expansion that requires O(n^d) operations, the system segments the problem into d independent linear projections, each computable in O(n) time, thereby reducing overall computational time while preserving accuracy.
3Productivity
If traditional Gaussian kernel approximations are used, then computation is simplified, but accuracy deteriorates because assumptions do not hold for polynomial kernels
Solution Approach 1:
The patent changes the fundamental parameters of the approximation approach by removing the positive definiteness constraint that Gaussian kernels require. The randomized feature maps can represent indefinite kernels, allowing accurate approximation of polynomial kernels without being constrained by Gaussian assumptions, thereby maintaining both computational efficiency and approximation accuracy.
Solution Approach 2:
The patent creates explicit feature maps that copy the exact properties of polynomial kernels including their indefinite nature, rather than copying Gaussian kernel properties. This allows the system to maintain polynomial kernel accuracy while achieving the computational simplicity of linear methods.
4Reliability
If explicit polynomial feature expansions are used, then kernel properties are preserved, but memory requirements become prohibitive
Solution Approach 1:
The patent changes the parameter representation from explicit polynomial coefficients to randomized feature map weights. This parameter transformation allows the system to represent polynomial kernel properties with fewer parameters, reducing memory requirements from exponential in polynomial degree to linear in feature map dimensionality.
Solution Approach 2:
The patent creates compact feature map representations that copy the essential kernel properties without storing full polynomial expansions. By using randomized projections, the system preserves kernel decision boundaries while using minimal memory, effectively copying only the necessary properties rather than storing complete polynomial representations.
Data Source
AI summary
Implementations provide for use of spherical random features for polynomial kernels and large-scale learning. An example method includes receiving a polynomial kernel, approximating the polynomial kernel by generating a nonlinear randomized feature map, and storing the nonlinear feature map. Generating the nonlinear randomized feature map includes determining optimal coefficient values and standard deviation values for the polynomial kernel, determining an optimal probability distribution of vector values for the polynomial kernel based on a sum of Gaussian kernels that use the optimal coefficient values, selecting a sample of the vectors, and determining the nonlinear randomized feature map using the sampled vectors. Another example method includes normalizing a first feature vector for a data item, transforming the first feature vector into a second feature vector using a feature map that approximates a polynomial kernel with an explicit nonlinear feature map, and providing the second feature vector to a support vector machine.