Query Result Sampling for Unbiased Search Engine Evaluation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale databases face challenges in delivering accurate and relevant data due to dynamic inventory and user-specific preferences, making it difficult to evaluate the performance of query engines effectively.
Innovation Solution
A process is introduced to sample historical query records, stratifying them based on search phrase frequencies to create a representative set that reduces bias and enhances training quality for item query engines.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all historical query records are used to evaluate query engine performance, then the evaluation comprehensiveness is improved, but the processing time and computational resources increase significantly
Solution Approach 1:
The patent segments the historical query records into different strata based on search phrase frequency categories (common, uncommon, rare). This segmentation allows the system to evaluate a representative sample from each stratum rather than processing all records uniformly, thereby reducing processing time while maintaining evaluation comprehensiveness across different query types.
Solution Approach 2:
The patent applies partial action by selecting a subset of historical query records for evaluation based on stratified sampling. Instead of evaluating all records, the system evaluates a carefully selected portion that represents the full distribution of query frequencies, achieving sufficient evaluation comprehensiveness with reduced processing time.
2Productivity
If random sampling is used on historical query records, then the processing efficiency is improved, but the evaluation accuracy deteriorates due to bias toward common search phrases
Solution Approach 1:
The patent divides the historical query records into distinct strata based on search phrase frequency (common, uncommon, rare). This segmentation ensures that each frequency category is independently represented in the sample, preventing the dominance of common phrases and maintaining evaluation accuracy while allowing efficient processing of a manageable sample size.
Solution Approach 2:
The patent applies local quality by ensuring that each stratum (frequency category) is adequately represented in the sample according to its specific characteristics. Rather than uniform random sampling, the system tailors the sampling approach to each local group (stratum), guaranteeing that rare and uncommon phrases receive appropriate representation proportional to their importance.
3Productivity
If the query engine is optimized for common search phrases, then the performance on frequent queries is improved, but the performance on rare and uncommon queries deteriorates
Solution Approach 1:
The patent segments the evaluation process into distinct frequency-based strata, allowing the system to assess and optimize performance across different query categories separately. This segmentation enables the query engine to maintain specialized optimization for common phrases while ensuring that rare and uncommon phrases receive adequate attention through targeted sampling and evaluation.
Solution Approach 2:
The patent changes the sampling parameters based on search phrase frequency characteristics. By adjusting the sampling strategy to account for frequency distribution, the system ensures that the query engine is evaluated and optimized across the full spectrum of query types, preventing over-optimization for common phrases at the expense of rare queries.
Data Source
AI summary
An online system may generate numerous search records in response to searches requested by users. The online system may use a specific way to sample the historical search records to reduce biases in sampling. For example, the online system retrieves historical query records associated with an item query engine. The set of historical query records includes a plurality of search phrases. A historical query record is associated with a search phrase and a list of items returned by the item query engine. The online system determines the search frequencies for the search phrases. The online system stratifies the historical query records into a plurality of bins according to the search frequencies of the search phrases. The online system samples the historical query records from the plurality of bins to collect a representative set of historical query records and outputs the representative set of historical query records for rating.