Mixed-Moment Embedding Loss for Independent Feature Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing self-supervised representation learning methods fail to effectively minimize redundancy among multiple feature variables beyond pairwise correlations, limiting the learning of meaningful and independent features.
Innovation Solution
A self-supervised representation learning (SSRL) circuitry that utilizes a normalizer and loss function to normalize feature variables using zero mean and unit standard deviation, and adjusts network parameters based on a factorizable mixed moment loss function to reduce total correlation among feature variables.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If existing self-supervised learning methods minimize pairwise covariance matrix to reduce feature redundancy, then pairwise correlation is reduced, but total correlation among multiple feature variables remains high
Solution Approach 1:
The patent extends the redundancy minimization from pairwise (2nd order) statistics to higher-order (Kth order, where K>2) mixed moments. This dimensional extension in the statistical space allows capturing multi-variable dependencies that pairwise correlations miss, thereby reducing total correlation while maintaining computational feasibility through factorization.
Solution Approach 2:
The patent changes the order parameter from 2 to K (where K>2) in the moment calculation. By adjusting this parameter, the method transitions from minimizing pairwise covariance to minimizing higher-order mixed moments, which captures more complex dependency structures and achieves better feature independence.
2Reliability
If higher-order mixed moments are used to minimize total correlation, then mutual independence is improved, but computational complexity increases
Solution Approach 1:
The patent segments the computation of higher-order mixed moments by factorizing them into products of lower-order moments. Specifically, the Kth-order mixed moment is decomposed into a product of (K-1)th-order moments, which are further decomposed recursively. This segmentation reduces computational complexity from O(N^K) to a manageable level while preserving the ability to capture multi-variable dependencies.
Solution Approach 2:
The patent computes only the necessary higher-order moments required for the loss function, rather than all possible moments. By selectively computing moments of order K where K>2 but not excessively large, the method achieves sufficient feature independence with controlled computational cost.
Data Source
AI summary
In an embodiment, there is provided a self-supervised representation learning (SSRL) circuitry. The SSRL circuitry includes a normalizer circuitry, and a loss function circuitry. The normalizer circuitry is configured to receive a number. T, batches of embedding features. Each batch includes a number. N, embedding features. The number N corresponds to a number of input samples in a training batch. The number T corresponds to a number of respective transformed batches. Each transformed batch corresponds to a respective transformation of the training batch. The embedding features may be related to the transformed batches. Each embedding feature has a dimension. D. and each embedding feature element corresponds to a respective feature variable. The normalizer circuitry is further configured to normalize each feature variable of a selected batch, using a zero mean and a unit standard deviation of the selected batch. A loss function circuitry is configured to determine a loss function based, at least in part, on a factorizable mixed moment of a plurality of normalized feature variables. The mixed moment is of order K. K is less than or equal to the embedding feature dimension D.