Ranking Function Evaluation Using Offline Exploration Logs

Solution Overview

Problem

Existing methods for evaluating ranking functions in information retrieval systems are costly and inefficient due to the need for extensive online A/B testing, which consumes resources and user time.

Innovation Solution

An offline evaluation system using production log data to generate an exploration data set, simulating randomized data collection through result diversity and causal inference techniques, allowing for the estimation of online metrics and identification of preferred ranking functions before live testing.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If online A/B testing is used to evaluate ranking functions, then measurement precision of ranking function performance is improved, but resource consumption and time cost increase significantly

Engineering Contradiction:

Improvemeasurement precisionVSAvoidtime cost

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The patent performs preliminary offline evaluation of ranking functions using historical production log data before conducting online A/B testing. By pre-processing production logs to generate exploration datasets and evaluating candidate ranking functions offline, the system identifies promising candidates that can then be tested online with reduced time cost and resource consumption.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent creates synthetic exploration datasets that replicate the characteristics of online user interactions by re-weighting and re-sampling historical production log data. This copying approach allows offline simulation of online ranking scenarios, enabling accurate performance estimation without actual user engagement during the evaluation phase.

Inventive Principle:

Principle #26Copying

2Measurement precision

If online A/B testing is used to evaluate ranking functions, then measurement precision is improved, but network bandwidth and system resources are consumed

Engineering Contradiction:

Improvemeasurement precisionVSAvoidnetwork bandwidth

Core Design Contradiction:

Measurement precisionVSLoss of energy

Solution Approach 1:

The patent creates synthetic exploration datasets that replicate the characteristics of online user interactions by re-weighting and re-sampling historical production log data. This copying approach allows offline simulation of online ranking scenarios, enabling accurate performance estimation without actual user engagement during the evaluation phase.

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The system uses its own historical production log data to conduct offline evaluations, making the evaluation process self-sufficient without requiring external user interactions or additional network resources. The production logs themselves serve as the evaluation dataset after appropriate processing.

Inventive Principle:

Principle #25Self-service

3Reliability

If extensive online testing is conducted to ensure high-quality search results, then reliability of ranking functions is improved, but productivity of the information retrieval system decreases

Engineering Contradiction:

ImprovereliabilityVSAvoidproductivity

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent performs preliminary offline evaluation of ranking functions using historical production log data before conducting online A/B testing. By pre-processing production logs to generate exploration datasets and evaluating candidate ranking functions offline, the system identifies promising candidates that can then be tested online with reduced time cost and resource consumption.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent applies partial online testing by conducting comprehensive offline evaluation first, then performing only necessary partial online A/B tests on selected candidates rather than exhaustive testing of all possible ranking functions. This partial action approach maintains reliability while improving productivity.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS12511299B2Offline evaluation of ranking functions

Publication Date: 2025.12.30 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

An example system includes a processor to receive production log data and pre-process the production log data to generate an exploration data set. The production log data includes continuous updates of a ranking function associated with different actions for a same query issued by a same user in different impressions. The processor can perform offline estimation of online metrics for ranking functions using the exploration data set, evaluate a proposed ranking function by comparing the estimated online metrics to a set of baseline metrics of a baseline ranking function, detect that the proposed ranking function is a preferred ranking function having a higher detected metric score than the baseline ranking function based on a comparison of the estimated online metrics with the baseline metrics of the baseline ranking function, and execute an action on an information retrieval system based on the preferred ranking function.