RFAD Algorithm Non-Deterministic Feature Approximation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current dataset distillation methods, such as Kernel-Inducing Points (KIP), are computationally expensive due to the exact calculation of the neural tangent kernel matrix, limiting their applicability and efficiency in summarizing large datasets into smaller, representative sets while maintaining accuracy.
Innovation Solution
The Random Feature Approximation Distillation (RFAD) algorithm employs a non-deterministic feature approximation of the neural network Gaussian process kernel, reducing kernel matrix computation complexity from O(|S|2) to O(|S|) and combining it with modified Platt scaling, enabling faster and more accurate dataset distillation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If exact calculation of the neural tangent kernel matrix is used in KIP algorithm, then dataset distillation accuracy is improved, but computational complexity increases to O(|S|2)
Solution Approach 1:
The patent replaces the expensive exact NTK calculation with a cheaper random feature approximation that can be discarded and regenerated. This approximation uses random projections to estimate kernel values without computing the full NTK matrix, reducing computational complexity from O(|S|2) to O(|S|) while maintaining acceptable distillation accuracy.
Solution Approach 2:
The patent changes the computational parameters by switching from exact kernel calculation to approximate kernel estimation using random features. This parameter change transforms the computational complexity from quadratic to linear in terms of coreset size, making the algorithm scalable to larger datasets while preserving the essential functionality of dataset distillation.
2Measurement precision
If exact calculation of the neural tangent kernel matrix is used in KIP algorithm, then dataset distillation accuracy is improved, but processing speed decreases significantly
Solution Approach 1:
The patent employs a computationally inexpensive random feature approximation that can be quickly computed and discarded. This approximation method generates fast estimates of kernel values without the heavy computational burden of exact NTK calculation, enabling rapid dataset distillation while maintaining reasonable accuracy through the use of random projections and feature mappings.
Solution Approach 2:
The patent applies partial action by using a simplified random feature approximation instead of the complete exact NTK calculation. This partial approach computes only the necessary kernel estimates using random features, sacrificing some precision for significant gains in processing speed, which is acceptable for many practical dataset distillation applications.
3Measurement precision
If KIP algorithm is used for dataset distillation, then accuracy is improved, but applicability is limited due to high computational cost
Solution Approach 1:
The patent changes the computational parameters by replacing the exact NTK calculation with a random feature-based approximation. This parameter change reduces the computational complexity from O(|S|2) to O(|S|), making the algorithm applicable to larger datasets and more resource-constrained environments while maintaining the core functionality of accurate dataset distillation.
Solution Approach 2:
The patent uses a computationally lightweight random feature approximation that can be applied broadly across different datasets and scenarios. This approximation method is inexpensive to compute and can be discarded after use, enabling the algorithm to be applied to a wider variety of practical problems including large-scale image datasets and resource-limited settings where the original KIP algorithm would be infeasible.
Data Source
AI summary
Dataset distillation compresses large datasets into smaller synthetic coresets that retain performance with the aim of reducing storage and computational burdens of processing an original, entire dataset. The present disclosure provides an improved algorithm that uses a non-deterministic feature approximation of neural network Gaussian process (NNGP) kernels, or other trained kernels, that reduces a kernel matrix computation to O(|S|). When combined with a modified Platt scaling loss, the disclosed algorithm can provide at least a 100-fold speedup over a Kernel-Inducing Points (KIP) algorithm and can run on a single graphics processing unit. The disclosed Random Feature Approximation Distillation (RFAD) algorithm can perform competitively with other dataset condensation algorithms in accuracy over a range of large-scale datasets, both in kernel regression and finite-width network training. The disclosed techniques can be effective on tasks such as model interpretability and data privacy preservation.