Automated Information Retrieval Evaluation Using Recall Metrics

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing information retrieval system evaluation methods rely heavily on manual evaluation, are limited in applicability, and lack accuracy in reflecting the retrieval performance of the system.

Innovation Solution

An automated method and device that collect behavior data samples, compute recall rates and correctness percentages by comparing evaluation retrieval results with sample retrieval results, and calculate an evaluation indicator using these metrics to assess the information retrieval system's performance.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If manual evaluation is used to assess information retrieval system performance, then evaluation accuracy can be maintained, but evaluation efficiency and automation level deteriorate

Engineering Contradiction:

Improveevaluation accuracyVSAvoidevaluation automation level

Core Design Contradiction:

Measurement precisionVSExtent of automation

Solution Approach 1:

The information retrieval system performs self-evaluation by automatically comparing its retrieval results against ground truth data and computing performance metrics such as precision, recall, and F1 score. This eliminates the need for manual evaluation while maintaining accurate measurement of retrieval quality.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent replaces manual evaluation processes with automated computational methods. The system automatically computes performance metrics through mathematical calculations comparing retrieval results with ground truth, substituting human evaluators with algorithmic assessment mechanisms.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Device complexity

If existing evaluation methods are used for non-addressing retrieval systems, then evaluation process remains simple, but evaluation accuracy and applicability deteriorate

Engineering Contradiction:

Improveevaluation process complexityVSAvoidretrieval performance accuracy

Core Design Contradiction:

Device complexityVSMeasurement precision

Solution Approach 1:

The patent introduces new evaluation parameters specifically tailored for non-addressing retrieval systems, including precision at top-k results, recall at top-k results, and F1 score. These parameters adapt the evaluation framework to accurately measure performance in scenarios where users do not specify precise location or address information.

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The evaluation method is designed to be universally applicable across different types of information retrieval systems, including both addressing and non-addressing retrieval scenarios. The same framework can evaluate various retrieval tasks by adjusting the ground truth data and performance metrics accordingly.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS9436729B2Information retrieval system evaluation method, device and storage medium

Publication Date: 2016.09.06 TENCENT TECHNOLOGY (SHENZHEN) CO LTD

AI summary

The present disclosure discloses an information retrieval system evaluation method, device and storage medium. A ratio between the sum of all the related object parameters of a keyword in keyword in an evaluation retrieval result set and the sum of all the related object parameters of the keyword in keyword in a retrieval result set is used to compute a recall rate of an information retrieval system. And the recall rate is introduced to evaluate the information retrieval system, thereby enhancing accuracy of quantitative evaluation of the information retrieval system, and improving the automation degree of evaluation.