Automated POI Query Generation for Search Quality Assessment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing search engines face challenges in accurately assessing the quality of search results, particularly for point-of-interest queries, due to the laborious and resource-intensive nature of manual testing and the limitations of using user logs, which often contain low-quality and ambiguous data, leading to increased costs and resource consumption.
Innovation Solution
An automated method and apparatus generate test queries using point-of-interest information and permutations thereof, employing word-level and character-level modifications to mimic user idiosyncrasies, and analyze responses to calculate quality metrics, thereby improving search engine performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual testing methods are used to assess search result quality, then measurement precision can be maintained through human evaluation, but productivity decreases due to laborious and resource-intensive processes
Solution Approach 1:
The system creates synthetic test queries by copying and modifying real user queries through word-level and character-level permutations. This generates large volumes of test data automatically, replacing manual query creation while maintaining the statistical properties needed for accurate quality assessment
Solution Approach 2:
The system performs self-testing by automatically generating test queries, executing searches, evaluating results against ground truth, and identifying quality issues without human intervention. This enables continuous automated quality monitoring that maintains precision while dramatically improving productivity
2Device complexity
If user logs are used for testing, then device complexity is reduced by using existing data sources, but measurement precision deteriorates due to low-quality and ambiguous data
Solution Approach 1:
The system introduces an intermediary processing layer that takes raw user logs and automatically generates refined test queries through systematic word-level and character-level modifications. This intermediary step cleans and enhances the data while maintaining the simplicity of using existing log sources
Solution Approach 2:
The system performs preliminary processing of user logs by pre-generating multiple query permutations and variations before actual testing. This prepares high-quality test data in advance, ensuring measurement precision is not compromised when using simple log-based data sources
3Productivity
If automated query generation is implemented, then productivity increases through faster and more scalable testing, but device complexity increases due to additional processing requirements
Solution Approach 1:
The system segments the query generation process into distinct modular components: word-level modification module, character-level modification module, permutation generation module, and quality evaluation module. This segmentation enables high-throughput automated testing while managing complexity through clear separation of concerns and reusable components
Data Source
AI summary
A method, apparatus and computer program product are provided for generating search engine input queries using point-of-interest information and permutations thereof to assess the quality of search results produced by and to improve the search engine. Methods may include: identifying a reference point-of-interest from a reference data database; extracting at least one point-of-interest attribute from the reference point-of-interest; identifying a geographic location of the reference point-of-interest; calculating a test query search center based on the geographic location of the reference point-of-interest; generating a test query based on the at least one point-of-interest attribute from the reference point-of-interest and the test query search center; querying a search engine using the test query; receiving one or more responses to the test query; analyzing the one or more responses to calculate response quality metrics; and modifying the search engine based at least in part on the metrics.