Attention-Matched Dataset Distillation Without Pretrained Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing dataset distillation methods face challenges with computational costs and biased representations due to reliance on pre-trained networks and bi-level optimization, limiting their scalability and performance, especially for large datasets like ImageNet-1K.
Innovation Solution
The proposed Dataset Distillation with Attention Matching (DataDAM) approach leverages spatial attention maps from randomly initialized neural networks to generate unbiased synthetic datasets by matching feature distributions across different layers, using a complementary loss to align real and synthetic data distributions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If pre-trained networks and bi-level optimization are used for dataset distillation, then model performance can be maintained, but computational costs and complexity increase significantly
Solution Approach 1:
The patent extracts and removes the pre-trained network requirement from the dataset distillation process. By using randomly initialized networks instead of pre-trained ones, the method eliminates the need for complex bi-level optimization while maintaining distillation effectiveness, thus reducing computational complexity without sacrificing performance
Solution Approach 2:
Instead of using pre-trained networks as the starting point (conventional approach), the patent inverts the approach by using randomly initialized networks. This inversion simplifies the optimization landscape and eliminates the need for bi-level optimization, resolving the contradiction between performance maintenance and complexity reduction
2Loss of information
If pre-trained networks are used for dataset distillation, then feature representations can be leveraged, but bias towards specific examples increases
Solution Approach 1:
The patent creates unbiased copies of the data distribution through randomly initialized networks. Instead of copying features from pre-trained networks (which introduce bias), the method uses multiple random networks to generate diverse, unbiased representations that collectively capture the true data distribution without favoring specific examples
Solution Approach 2:
The patent changes the initialization parameters from pre-trained weights to random weights. This parameter change eliminates the bias embedded in pre-trained networks while maintaining the ability to learn effective feature representations through the distillation process
3Reliability
If full training sets are used, then data distribution and classification utility are maintained, but computational costs and training time increase significantly
Solution Approach 1:
The patent enables the synthetic dataset to serve itself as a complete training replacement. By carefully distilling the essential distributional properties into the synthetic data, the method allows the condensed dataset to stand alone without requiring the original full training set, thus achieving both distribution fidelity and training efficiency
4Productivity
If smaller synthetic datasets are generated, then training speed and memory efficiency improve, but performance degradation occurs due to distribution issues
Solution Approach 1:
The patent transitions from matching distributions in the input data space to matching distributions in the network output space (logits space). This dimensional transformation allows small synthetic datasets to capture essential distributional properties that directly translate to classification performance, resolving the trade-off between dataset size and performance
Data Source
AI summary
Systems and methods proposed herein are directed to an improved approach and corresponding data architecture for generating condensed synthetic sets using a dataset distillation with attention matching (DataDAM) approach that matches spatial attention maps of real and synthetic data generated by different layers within a family of randomly initialized neural networks.