Ad Conversion Model Distillation for Seasonal Pattern Retention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional systems for training ad machine learning models using a sliding window approach lead to overfitting and catastrophic forgetting, neglecting long-term patterns and seasonal trends, which are crucial for optimizing ad performance in dynamic markets.
Innovation Solution
Implement a knowledge distillation approach where past models act as 'experts' to generate soft labels for new models, and a dual-model training system is used with one model trained on new data and another on both new and historical data, ensuring a balanced understanding of current and historical insights.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a sliding window approach is used for incremental model training, then the model can adapt to new data efficiently, but it leads to catastrophic forgetting of historical patterns and seasonal trends
Solution Approach 1:
The patent creates multiple copies of the neural network model, each trained on different time periods (e.g., Model 1 on January data, Model 2 on February data). These model copies preserve historical patterns without interfering with each other, allowing the system to retain seasonal trends while adapting to new data through the ensemble approach.
Solution Approach 2:
The patent merges predictions from multiple model copies through averaging or voting mechanisms. This combination allows the system to benefit from both historical patterns (captured by older models) and recent trends (captured by newer models), resolving the contradiction between adapting to new data and retaining historical information.
2Productivity
If only recent data is used for training, then the model responds quickly to current market conditions, but it overfits to short-term patterns and loses generalization ability
Solution Approach 1:
The patent segments the training data into multiple time-period-specific datasets and trains separate model copies on each segment. This segmentation allows each model to be trained efficiently on its specific period while the ensemble provides overall generalization, balancing training speed with reliability.
Solution Approach 2:
The patent adds a temporal dimension to the model architecture by creating model copies across different time periods. This dimensional expansion allows the system to maintain fast training on individual periods while achieving robust generalization through the multi-temporal ensemble, effectively moving from a single-model timeline to a multi-model temporal landscape.
3Measurement precision
If the model is continuously retrained with new data, then it stays updated with current trends, but it forgets long-term seasonal patterns and catastrophic forgetting occurs
Solution Approach 1:
The patent performs preliminary training of multiple model copies on historical data before deployment. Each model copy is pre-trained on its specific time period's data, preserving seasonal patterns in advance. During operation, the ensemble of pre-trained models maintains long-term pattern retention while adapting to current trends through their collective predictions.
Data Source
AI summary
Described is a system for knowledge distillation in ad machine learning models by training a plurality of machine learning models on respective datasets collected over respective time periods; collecting a first dataset comprising ad impression data and ad conversion data over a first time period; applying each of the first dataset to the plurality of machine learning models to generate a plurality of labels; derive a value based on the plurality of labels; training a first machine learning model based on the application of the first dataset to the first machine learning model; applying a plurality of ads to the trained first machine learning model to receive individual predicted ad conversion rates for each of the plurality of ads; and ranking the ads based on the predicted ad conversion rates.