Neural Network Feature Compression via Entropy Penalty
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-dimensional features generated by pre-trained neural networks, such as CNNs, are expensive to store and traditional compression methods introduce distortions that reduce their effectiveness.
Innovation Solution
A method that optimizes the compressibility of features by identifying the feature layer as a bottleneck, introducing a penalty to balance compressibility and informativeness, using an entropy term in the loss function to enable efficient compression of neural network weights, and applying uniform noise during training to simulate quantization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional entropy based compression methods are applied to high-dimensional features, then storage space is reduced, but feature quality and effectiveness are distorted and reduced
Solution Approach 1:
The patent changes the parameter space by transforming high-dimensional continuous features into lower-dimensional discrete latent representations through learned compression functions. This parameter transformation enables efficient compression while preserving essential feature information, resolving the contradiction between storage reduction and quality maintenance.
Solution Approach 2:
The patent introduces an intermediary latent representation layer that acts as a bridge between the original high-dimensional features and the compressed storage format. This intermediary structure enables lossless or near-lossless compression by capturing essential information in a compact form, avoiding the quality degradation caused by traditional compression methods.
2Loss of information
If high-dimensional features are stored without compression, then feature informativeness is preserved, but storage costs become prohibitive
Solution Approach 1:
The patent extracts only the essential and discriminative information from high-dimensional features through learned compression, discarding redundant components. This extraction process reduces storage requirements while maintaining the informative content necessary for downstream tasks, resolving the contradiction between information preservation and storage efficiency.
Solution Approach 2:
The patent transforms the feature representation parameters from high-dimensional continuous space to low-dimensional discrete latent space, enabling compact storage. This parameter change maintains the essential informative content while dramatically reducing storage costs, making large-scale feature storage feasible.
3Productivity
If feature dimensionality is reduced for compression, then storage efficiency improves, but predictive power may be sacrificed
Solution Approach 1:
The patent employs feedback mechanisms during the compression learning process, where the compressed latent representations are evaluated based on their predictive performance on downstream tasks. This feedback guides the optimization of compression functions to preserve task-relevant information, ensuring that storage efficiency gains do not compromise predictive power.
Solution Approach 2:
The patent implements dynamic adaptation in the compression process, where the compression function learns to preserve different amounts of information depending on the specific task requirements. This dynamic approach allows the system to maintain predictive power by adaptively retaining task-critical features while compressing less important dimensions.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for receiving, by a neural network (NN), a dataset for generating features from the dataset. A first set of features is computed from the dataset using at least a feature layer of the NN. The first set of features i) is characterized by a measure of informativeness; and ii) is computed such that a size of the first set of features is compressible into a second set of features that is smaller in size than the first set of features and that has a same measure of informativeness as the measure of informativeness of the first set of features. The second set of features if generated from the first set of features using a compression method that compresses the first set of features to generate the second set of features.