Question-answering model training via consistency evaluation

Solution Overview

Problem

Conventional question-answering learning methods require extensive manual processing to ensure accuracy, leading to increased labor and inconsistent results due to the need for large volumes of manually-classified sentences.

Innovation Solution

A question-answering learning method and system that selectively identifies a small number of unlabeled sentences for manual labeling based on consistency and complementarity evaluation, using a classifier generation module, consistency evaluation module, and complementarity evaluation module to re-create and enhance question-answering models, reducing manual labor and improving accuracy.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If a large volume of manually-classified sentences is used to establish question-answering models, then the question-answering accuracy can be improved, but the manual processing labor increases significantly

Engineering Contradiction:

Improvequestion-answering accuracyVSAvoidmanual processing labor

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system enables self-service by allowing the question-answering model to automatically select and label sentences that would be most beneficial for its own improvement. The model identifies sentences with high uncertainty or low consistency scores and automatically processes them, eliminating the need for extensive manual labeling while continuously improving accuracy

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system implements feedback mechanisms where the question-answering model's performance is continuously evaluated, and the results are used to guide subsequent data selection and model retraining. The model receives feedback on its own performance metrics and uses this information to prioritize which sentences to process next, creating a closed-loop system that improves efficiency

Inventive Principle:

Principle #23Feedback

2Measurement precision

If new incoming sentences are falsely classified and correct labels are obtained by human labeling, then the question-answering accuracy can be improved, but the manual processing labor increases and results are not consistently increased

Engineering Contradiction:

Improvequestion-answering accuracyVSAvoidconsistency of results

Core Design Contradiction:

Measurement precisionVSReliability

Solution Approach 1:

The system enables self-service by allowing the question-answering model to automatically select and label sentences that would be most beneficial for its own improvement. The model identifies sentences with high uncertainty or low consistency scores and automatically processes them, eliminating the need for extensive manual labeling while continuously improving accuracy

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system implements feedback mechanisms where the question-answering model's performance is continuously evaluated, and the results are used to guide subsequent data selection and model retraining. The model receives feedback on its own performance metrics and uses this information to prioritize which sentences to process next, creating a closed-loop system that improves consistency

Inventive Principle:

Principle #23Feedback

3Measurement precision

If classifiers are re-created according to N1 labeled sentences and N4 selected to-be-labeled sentences, then the question-answering accuracy is stabilized, but the processing time increases

Engineering Contradiction:

Improvequestion-answering accuracy stabilityVSAvoidprocessing time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system extracts only the most critical information by selecting a small subset of N4 sentences that are most complementary to the existing N1 labeled sentences. Rather than retraining on all available data, the system identifies and processes only the sentences that will provide the maximum improvement, significantly reducing processing time while maintaining accuracy stability

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The system changes parameters by dynamically adjusting which sentences are selected for retraining based on consistency scores and complementarity metrics. The selection criteria and weighting parameters are modified based on the model's current performance state, allowing the system to adapt the retraining process to minimize processing time while maintaining accuracy

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS12014284B2Question-answering learning method and question-answering learning system using the same and computer program product thereof

Publication Date: 2024.06.18 IND TECH RES INST

AI summary

A question-answering learning method including the following steps is provided. Firstly, several classifiers are created according to N1 labeled sentences among N sentences. Then, at least one corresponding sentence type of each of the N2 unlabeled sentences among the N sentences is determined by each classifier. Then, N3 sentences are selected from the N2 unlabeled sentences according to a degree of consistency of determined results of the classifiers, wherein the determined results of the N3 sentences are determined by the classifiers and are inconsistent. Then, N4 mutually complementary sentences are selected as to-be-labeled sentences from the N3 sentences. Then, after the N4 selected to-be-labeled sentences are labeled, several classifiers are re-created according to the N1 labeled sentences and the N4 selected to-be-labeled sentences. Then, at least one of the previously created classifiers is added to the currently created classifiers to be members of the classifiers.