Delayed Batch Processing for Auction Over-Delivery Classification

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Auction experiments in content delivery systems often suffer from interference bias and over-delivery issues due to improper budget consumption and pacing, leading to misestimation of treatment effects and wasteful use of computing resources.

Innovation Solution

Implementing a delayed processing method that classifies over-delivered campaigns and adjusts auction experiment designs to prevent over-delivery by using a budget-split or quota-split design to circumvent interference bias and optimize campaign participation in auctions.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If auction experiments are conducted with standard budget consumption, then treatment effect estimation is performed, but interference bias occurs and treatment effects are misestimated

Engineering Contradiction:

Improvetreatment effect estimation accuracyVSAvoidexperiment validity

Core Design Contradiction:

Measurement precisionVSReliability

Solution Approach 1:

The patent segments the campaign budget into multiple partitions, where each partition is independently consumed by content requests randomized to different experiment arms. This segmentation prevents interference bias by ensuring that treatment and control groups do not compete for the same budget resources, thereby improving both measurement precision and experiment validity.

Inventive Principle:

Principle #1Segmentation

2Productivity

If campaigns are allowed to participate in auctions without delayed processing, then real-time auction execution is maintained, but over-delivery occurs and computing resources are wasted

Engineering Contradiction:

Improveauction execution speedVSAvoidcomputing resource waste

Core Design Contradiction:

ProductivityVSLoss of energy

Solution Approach 1:

The patent implements delayed processing that waits for a predetermined time period to elapse before determining whether a campaign is over-delivered. This preliminary waiting period allows the system to collect sufficient data to accurately assess delivery status, preventing premature conclusions and unnecessary computational waste while maintaining real-time auction execution.

Inventive Principle:

Principle #10Preliminary action

3Loss of energy

If budget pacing is applied to prevent over-delivery, then resource utilization is optimized, but treatment effect estimation becomes inaccurate due to interference bias

Engineering Contradiction:

Improvecomputing resource efficiencyVSAvoidtreatment effect accuracy

Core Design Contradiction:

Loss of energyVSMeasurement precision

Solution Approach 1:

The patent applies segmentation by dividing the budget into separate partitions for different experiment arms. This allows independent budget consumption without interference, enabling accurate treatment effect estimation while maintaining efficient resource utilization through controlled budget pacing in each partition.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces randomized assignment as an intermediary mechanism that distributes content requests to different experiment arms in a controlled manner. This intermediary process ensures that budget pacing does not create interference bias, as the randomization decouples the relationship between budget consumption and treatment assignment.

Inventive Principle:

Principle #24Intermediary (Mediator)

4Measurement precision

If delayed processing window is extended, then over-delivery detection accuracy is improved, but processing time increases

Engineering Contradiction:

Improveover-delivery detection accuracyVSAvoidprocessing delay

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The patent implements a predetermined time period for delayed processing that may be shorter than the full duration needed for complete data collection. This partial action approach balances detection accuracy with processing efficiency, obtaining sufficient data to make reliable over-delivery determinations without incurring excessive delays.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS11238488B2Delayed processing for over-delivery determination for content delivery system experimentation

Publication Date: 2022.02.01 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

A delayed grouping (batch) processing of previous campaign delivery pacing decisions and corresponding outcomes (deliveries) is used to configure a new auction experiment iteration. In the new iteration, a campaign that was previously over-delivered is classified as either (a) over-delivered due to incorrect pacing or (b) over-delivered due to auction experiment design. After the delayed processing, the new auction experiment iteration is conducted with a mitigating action taken on the previously over-delivered campaign if the campaign is classified as (b) over-delivered due to auction experiment design. For example, the mitigating action can include removing the campaign from a subsequent iteration of the experiment, or the experiment can be redesigned. By doing so, the over-delivery caused by the campaign due to the auction experiment design is avoided when performing the new auction experiment iteration.