Budget-Constrained Deep Q-Network for Dynamic Ad Campaign Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for dynamic campaign allocation in computational advertising fail to optimally allocate advertisement campaigns within a fixed budget, leading to suboptimal conversion rates and inefficient budget utilization, as they either rely on heuristic approaches or are not generalizable due to the use of linear programming.
Innovation Solution
A budget-constrained deep Q-network (DQN) and cumulative reward model are jointly trained to determine optimal campaign allocation policies, using a Markov Decision Process framework that considers user and advertiser states, with the DQN employing an epsilon greedy policy and the cumulative reward model rewarding maximum total conversions over an advertising duration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If linear programming is used for budget-constrained advertising allocation, then optimal policy can be determined, but the approach is prone to errors and not generalizable
Solution Approach 1:
The patent replaces the mechanical linear programming system with a neural network-based deep Q-learning system. The DQN learns optimal allocation policies through reinforcement learning from simulated environments, substituting the rigid mathematical optimization approach with a flexible neural network that can generalize to different scenarios without requiring manual formulation of objective functions and constraints.
Solution Approach 2:
The patent changes the fundamental parameters of the optimization approach by transitioning from explicit linear programming parameters (objective functions, constraints) to implicit neural network parameters (weights, biases). The DQN learns optimal policies by adjusting its internal parameters through gradient descent and reinforcement learning, enabling both optimality and generalizability simultaneously.
2Ease of operation
If static budget allocation is used at the start of advertising duration, then budget distribution is simplified, but optimal campaign allocation for maximizing conversions is not achieved
Solution Approach 1:
The patent introduces dynamics into the budget allocation system by using a Deep Q-Network that can adaptively adjust campaign allocations in real-time based on current state observations. The system transitions from static pre-defined allocation to dynamic learning-based allocation, where the DQN continuously learns optimal policies through interaction with the environment and updates its strategy accordingly.
Solution Approach 2:
The system employs self-service through autonomous learning where the DQN automatically determines optimal allocation policies without manual intervention. The neural network learns from simulated advertising scenarios and autonomously improves its allocation strategy, eliminating the need for continuous human optimization while achieving high conversion rates.
3Productivity
If deep learning is used for optimal allocation, then conversion maximization is achieved, but the approach does not consider budget constraints dynamically
Solution Approach 1:
The patent creates a universal Deep Q-Network that simultaneously handles both conversion maximization and dynamic budget constraint satisfaction. The DQN is designed to process multiple objectives (conversions and budget usage) through its state representation and reward function, making it a multi-functional system that can adapt to different budget scenarios and campaign types without requiring separate models.
Data Source
AI summary
In the world of digital advertising, optimally allocating an advertisement campaign within a fixed pre-defined budget for an advertising duration aimed at maximizing number of conversions is very important for an advertiser. Embodiments of present disclosure provides a robust and easily generalizable method of optimal allocation of advertisement campaign by formulating it as a constrained Markov Decision Process (MDP) defined by agent state comprising user state and advertiser state, action space comprising a plurality of ad campaigns, state transition routine and a cumulative reward model which rewards maximum total conversions in an advertising duration. The cumulative reward model is trained in conjunction with a deep Q-network for solving the MDP to optimally allocate advertisement campaign for an advertising duration within a constrained budget.