Contrastive Dual Gating for Sparse Feature Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current self-supervised learning methods require large and wide models for high accuracy, leading to high computation costs and inefficiencies, especially when trying to learn sparse features from scratch, and existing dynamic pruning algorithms struggle to effectively reduce computation without degrading accuracy.
Innovation Solution
The Contrastive Dual Gating (CDG) algorithm, which dynamically prunes uninformative features during contrastive learning by using spatial gating functions to learn sparse features in both contrastive branches, reducing computation without affecting trainability, and achieves higher accuracy and computation reduction compared to state-of-the-art methods.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If wide and deep models are used for self-supervised learning, then accuracy is improved, but computation cost increases
Solution Approach 1:
The model computation is segmented into base path and conditional path. The base path processes all input data through the full network, while the conditional path selectively processes only certain features based on learned importance scores. This segmentation allows the model to achieve high accuracy on important features while reducing computation on less important ones.
Solution Approach 2:
The system dynamically adjusts which features are processed through the conditional path based on learned importance scores. Unlike static pruning methods, the conditional path computation is dynamically activated or deactivated for different features based on their importance, allowing the model to adapt computation resources to actual needs during inference.
2Use of energy by moving object
If dynamic computation skipping is applied, then computation cost is reduced, but accuracy degrades
Solution Approach 1:
The system uses importance scores computed from the base path output as feedback to determine which features should be processed by the conditional path. This feedback mechanism ensures that features deemed important by the base path are further refined by the conditional path, while less important features are skipped, thereby maintaining accuracy while reducing computation.
Solution Approach 2:
Different features are treated with different levels of processing quality. Features with high importance scores receive full processing through the conditional path, while features with low importance scores are skipped. This local differentiation of processing quality ensures that computational resources are focused on features that matter most for accuracy.
3Quantity of substance
If static weight pruning is applied, then model size is reduced, but adaptability decreases
Solution Approach 1:
The system maintains full model weights but dynamically selects which weights to activate during inference through the conditional path mechanism. This dynamic approach preserves the adaptability of the full model while achieving effective pruning, as the model can adapt its computation pattern based on input characteristics rather than being fixed to a pruned structure.
Data Source
AI summary
A method of training a machine learning algorithm comprises providing a set of input data, performing transforms on the input data to generate augmented data, to provide transformed base paths into machine learning algorithm encoders, segmenting the augmented data, calculating main base path outputs by applying a weighting to the segmented augmented data, calculating pruning masks from the input and augmented data to apply to the base paths of the machine learning algorithm encoders, the pruning masks having a binary value for each segment in the segmented augmented data, calculating sparse conditional path outputs by performing a computation on the segments of the segmented augmented data, and calculating a final output as a sum of the main base path outputs and the sparse conditional path outputs. A computer-implemented system for learning sparse features of a dataset is also disclosed.