Copernican Loss for Discriminative Cosine Embedding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep neural network loss functions, such as Softmax loss, struggle to efficiently learn discriminative features in scenarios with a large number of samples per class, as they require convergence in both angle and norm, leading to inefficiencies and increased computational complexity, especially when using l2 embeddings.
Innovation Solution
The Copernican Loss (LC) augments the Softmax loss by minimizing intra-class variation and maximizing inter-class variation using cosine distance, eliminating the need for hard sample mining and data augmentation, and computes gradients efficiently using stochastic gradient descent on random mini-batches, focusing on cosine embeddings that utilize model complexity for angular margins.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If l2 embeddings are used to minimize intra-class variation, then features converge toward class centers, but the network must converge both angle and norm which is too difficult when samples per class is high
Solution Approach 1:
The patent extracts the norm constraint from the training objective, removing it from the loss function while maintaining it only during testing. This separates the angle-learning task from the norm-constraint task, making training more feasible while preserving the desired testing behavior through explicit L2 normalization of features before cosine distance computation.
Solution Approach 2:
The patent changes the embedding space from l2 (Euclidean) to cosine space by modifying the loss function to operate on angular margins rather than Euclidean distances. This parameter transformation allows the network to learn discriminative features through angle optimization alone, avoiding the difficulty of simultaneous angle and norm convergence.
2Reliability
If Softmax loss is used with cross-entropy, then gradients are well-behaved, but discrimination is limited compared to angular margin approaches
Solution Approach 1:
The patent merges the benefits of Softmax loss (well-behaved gradients through cross-entropy) with angular margin discrimination (through cosine distance computation). The combined approach uses cross-entropy to stabilize gradient flow while cosine distance provides superior angular discrimination, achieving both gradient reliability and high discrimination precision.
3Productivity
If cosine distance is used for testing, then norm constraints are ignored, but training with l2 embeddings creates a disconnect between training and testing behavior
Solution Approach 1:
The patent extracts the norm constraint from the training objective, removing it from the loss function while maintaining it only during testing. This separation resolves the training-testing disconnect by allowing flexible angle-based learning during training while applying norm normalization only when computing cosine distances during testing, making the system both efficient and consistent.
4Manufacturing precision
If COCO loss is used to maximize inter-class variation with class centers, then discrimination improves, but computational complexity increases significantly
Solution Approach 1:
The patent implements self-service by having each class maintain its own center (Planet center) that automatically adapts to the distribution of its samples. This eliminates the need for complex pair-wise gradient computations between all samples and all class centers, as each sample only needs to compute distance to its own class center and the global batch center, significantly reducing computational complexity while maintaining discrimination precision.
Data Source
AI summary
During training of deep neural networks, a Copernican loss (LC) is designed to augment a primary loss function, for example, a standard Softmax loss, to explicitly minimize intra-class variation and simultaneously maximize inter-class variation. Copernican loss operates using the cosine distance and thereby affects angles leading to a cosine embedding, which removes the disconnect between training and testing.