Search Algorithm Evaluation via Preliminary Environment Testing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing comparison tests for search engine algorithm upgrades are inefficient, particularly when a defective new algorithm is used, requiring extensive re-execution and modification, which is not preferable.
Innovation Solution
An information processing device and method that conducts a comparison test between a current and a new search algorithm using a preliminary environment, evaluating their performance based on relative evaluation values calculated from ranking information and relevance scores, allowing for a simplified acceptance-or-rejection judgment of the new algorithm's minimum search performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a new search algorithm is used for comparison test under actual environment, then the evaluation of contribution to sales (CTR, CVR) can be confirmed, but the test execution count increases and efficiency decreases
Solution Approach 1:
The patent applies preliminary action by conducting a preliminary comparison test in a preliminary environment before the actual environment test. This preliminary test evaluates basic search performance using automated evaluation indices, allowing defective algorithms to be filtered out early without consuming actual user traffic, thus improving test efficiency while maintaining evaluation reliability
2Reliability
If the new algorithm is defective in basic performance, then algorithm modification is required, but re-execution of the entire test process increases time consumption
Solution Approach 1:
The patent segments the evaluation process into two independent stages: (1) preliminary comparison test in preliminary environment evaluating basic search performance, and (2) actual environment test evaluating contribution to sales. This segmentation allows defective algorithms to be identified and modified after only the first stage, avoiding re-execution of the entire test process and reducing time loss
3Productivity
If a preliminary environment test is conducted before actual environment test, then defective algorithms can be filtered early, but the overall test process becomes more complex
Solution Approach 1:
The patent introduces a preliminary environment as an intermediary between algorithm development and actual environment testing. This preliminary environment uses automated evaluation indices as a mediator to objectively assess basic search performance, simplifying the overall process by replacing subjective manual evaluation while improving test efficiency
Data Source
AI summary
Under a preliminary environment, test searches with a current algorithm and a new algorithm are performed. Based on evaluation values related to rankings of the searches calculated based on these results, a first evaluation value for the current algorithm and a second evaluation value for the new algorithm are acquired. A control is performed to present relative evaluation information of the new algorithm with respect to the current algorithm based on these first evaluation value and second evaluation value. The evaluation values related to the rankings are calculated based on relevance corresponding information. In the relevance corresponding information, a relevance score is made to correspond to a test query of a test content data item for each set of the test query and the test content data item related to this test query.