Convex Feature Normalization for Deep Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deep learning methods for supervised classification, particularly in face recognition, face challenges due to non-convex formulations in feature normalization, leading to local minima and imbalanced angular classification margins, as well as a disconnect between training and testing protocols.
Innovation Solution
The introduction of Ring Loss, a convex feature normalization method that learns to constrain features to a scaled unit circle, allowing for robust and balanced feature representation by augmenting the primary loss function, thereby maintaining convexity and improving optimization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If direct normalization is applied in deep networks with non-convex formulation, then feature normalization is achieved, but local minima are generated by the loss function
Solution Approach 1:
The patent changes the formulation approach from direct hard normalization to soft normalization through a convex loss function. By parameterizing the normalization constraint as a continuous optimization objective rather than a discrete operation, the method transforms the non-convex problem into a convex one, eliminating local minima while maintaining feature normalization effects
Solution Approach 2:
The patent replaces the mechanical hard normalization operation (projection onto unit sphere) with a mathematical convex optimization mechanism. Instead of forcibly constraining features to unit norm, the system uses a differentiable convex loss function that guides features toward normalized states through gradient descent, substituting a rigid mechanical constraint with a flexible mathematical optimization process
2Measurement precision
If Softmax loss is used with norm constraints, then classification performance is improved, but the formulation becomes non-convex in feature representations
Solution Approach 1:
The patent merges the classification loss and normalization constraint into a single unified convex loss function. Rather than applying Softmax loss separately and then enforcing norm constraints, the method combines both objectives into one differentiable convex formulation that simultaneously optimizes classification accuracy and feature normalization, simplifying the overall optimization landscape
Solution Approach 2:
Instead of applying normalization as a post-processing step or separate constraint after Softmax loss, the patent inverts the approach by embedding normalization directly into the loss function itself. The convex normalization loss is applied during the forward pass and backpropagated through gradients, making normalization an intrinsic part of the optimization process rather than an external constraint
3Manufacturing precision
If feature normalization is applied during training, then angular classification margins are balanced, but a gap exists between training and testing protocols
Solution Approach 1:
The patent implements self-service normalization where the network learns to produce normalized features automatically through the convex loss function during training. Since the loss function inherently guides features toward unit norm, the network becomes self-capable of producing normalization-compatible features without requiring separate testing-time normalization operations, eliminating the protocol gap
Solution Approach 2:
The patent performs preliminary normalization action during the training phase by incorporating the convex normalization loss into the training objective. This preliminary constraint ensures that features are already normalized when the model is deployed, so no additional normalization step is needed during testing, thereby aligning training and testing protocols
Data Source
AI summary
A method of training a deep neural network, such as would be used in facial recognition, includes training the deep neural network to normalize feature vectors to a learned value representing a radius of a multi-dimensional hypersphere using a convex augmentation of the primary loss function.