Feature-Wise Embedding Normalization for CTR Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing normalization methods for neural network models, such as batch normalization (BN) and layer normalization (LN), do not effectively preserve the importance of feature embeddings, leading to reduced accuracy in click-through rate (CTR) prediction models.
Innovation Solution
A method for training neural network models that normalizes embedding vectors using feature-wise linear transformation parameters, including a scale and shift parameter, which are applied uniformly to all elements of the embedding vector, allowing the model to reflect the importance of individual features.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Stability of the object's composition
If batch normalization (BN) or layer normalization (LN) is applied to stabilize element training, then training stability is improved, but the importance of feature embedding is lost leading to reduced prediction accuracy
Solution Approach 1:
The patent applies different normalization strategies to different parts of the feature processing pipeline. Specifically, it normalizes feature embeddings individually before processing, while applying normalization layers selectively to certain neural network layers. This local differentiation preserves the importance characteristics of individual feature embeddings while still providing stability where needed, resolving the contradiction between training stability and prediction accuracy.
Solution Approach 2:
The patent introduces learnable scaling and shifting parameters (gamma and beta) in the normalization layers, allowing the model to adaptively adjust the normalized feature distributions during training. This parameter adjustment mechanism enables the system to maintain both training stability through normalization and prediction accuracy through learned parameter optimization, directly addressing the contradiction.
2Device complexity
If constant parameters are used in normalization for all elements, then computational simplicity is maintained, but feature importance cannot be reflected
Solution Approach 1:
The patent segments the normalization process into two distinct stages: (1) individual feature embedding normalization that preserves importance characteristics, and (2) selective layer normalization with learnable parameters that provides stability. This segmentation allows each part to serve its specific function without compromising the other, maintaining computational efficiency while preserving feature importance.
Solution Approach 2:
The patent transitions from static constant parameters to dynamic learnable parameters (gamma and beta) that are optimized during training. This dynamic adjustment allows the normalization process to adapt to the specific characteristics of different features and layers, preserving feature importance while maintaining computational tractability through gradient-based optimization.
Data Source
AI summary
A method of training a neural network model for predicting a click-through rate (CTR) of a user in an electronic device includes normalizing an embedding vector on the basis of a feature-wise linear transformation parameter, and inputting the normalized embedding vector into a neural network layer, wherein the feature-wise linear transformation parameter is defined such that the same value is applied to all elements of the embedding vector.