Entropy Minimization for Contrastive Learning Robustness

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Contrastive learning lacks a theoretical basis, leading to models with limited generalization ability and robustness, as it primarily aims for better hidden space representation rather than discriminant representation for downstream tasks.

Innovation Solution

A new contrastive learning framework inspired by the phenomenon of water-oil separation, where a model is trained by minimizing entropy information using positive and negative sample sets to generate features, constructing a similarity matrix and updating the model based on entropy loss, enhancing feature representation, generalization, and robustness.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If contrastive learning is used to train models, then hidden space representation is improved, but generalization ability and robustness deteriorate

Engineering Contradiction:

Improvehidden space representationVSAvoidgeneralization ability and robustness

Core Design Contradiction:

Measurement precisionVSReliability

Solution Approach 1:

The patent segments the training process into two distinct phases: contrastive learning for hidden space representation and entropy minimization for discriminant representation. This segmentation allows each phase to optimize for its specific goal without compromising the other, resolving the contradiction between representation quality and generalization ability

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces entropy as a new parameter to optimize during training, shifting from purely contrastive loss to entropy-minimized representation. This parameter change enables the model to achieve both good hidden space representation and improved generalization by controlling the information distribution in the feature space

Inventive Principle:

Principle #35Parameter changes

2Loss of information

If contrastive learning focuses on hidden space representation, then feature similarity is improved, but discriminant representation for downstream tasks deteriorates

Engineering Contradiction:

Improvefeature similarityVSAvoiddiscriminant representation

Core Design Contradiction:

Loss of informationVSAdaptability or versatility

Solution Approach 1:

Instead of directly optimizing for discriminant representation, the patent inverts the approach by minimizing entropy to achieve uniform information distribution. This inverted strategy naturally leads to better discriminant representation while preserving feature similarity, as uniform entropy encourages diverse yet informative feature embeddings

Inventive Principle:

Principle #13The other way round (Inversion)

Data Source

PatentUS20230129870A1Method, apparatus, electronic device and medium for training models

Publication Date: 2023.04.27 EMC IP HLDG CO LLC

AI summary

Embodiments of the present disclosure provide a method and an apparatus for training a model, an electronic device, and a medium. This method includes: generating a first group of features and a second group of features respectively from a first sample set and a second sample set based on the model, wherein the first sample set is of a first category, and the second sample set is of a second category different from the first category; generating a first similarity matrix for the first sample set and the second sample set based on the first group of features and the second group of features; determining a first loss for the first sample set and the second sample set based on the first similarity matrix; and updating the model based on the first loss.