Corpus Cleaning via Context-Aware Information Amount Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Human-computer dialogue systems face challenges in generalization due to sparse training utterances, where the model focuses on irrelevant lexical features, leading to incorrect intention identification, and current solutions like stop word lists are not context-specific and flawed.
Innovation Solution
A corpus cleaning method using a pre-trained general model to predict the information amount of each word in an utterance, determining and removing redundant words based on context, ensuring accurate intention classification by setting a predicted value threshold.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If sparse training utterances are used, then the corpus size is reduced and entry efficiency is improved, but the model focuses on irrelevant lexical features and generalization ability deteriorates
Solution Approach 1:
The patent extracts and removes stop words from training utterances using a pre-trained language model to predict word importance. By taking out irrelevant lexical features (stop words) that harm generalization, the system maintains high entry efficiency while improving intention identification accuracy through cleaner training data
Solution Approach 2:
The patent introduces a pre-trained language model as an intermediary to evaluate and filter words in the corpus. This mediator predicts the importance of each word and identifies stop words that should be removed, resolving the contradiction between keeping concise corpora and maintaining generalization ability
2Productivity
If a stop word list is used to remove irrelevant words, then processing speed is improved, but semantic parsing accuracy deteriorates due to context-insensitive removal
Solution Approach 1:
The patent replaces static stop word lists with a dynamic filtering mechanism using a pre-trained language model. The model adaptively evaluates each word's importance based on contextual information, allowing the system to maintain high processing speed while accurately preserving semantically important words that would otherwise be incorrectly removed
Solution Approach 2:
The patent changes the parameter of word importance evaluation from fixed (pre-defined stop word lists) to variable (context-dependent predictions by language model). This allows the system to dynamically adjust which words to remove based on their contextual significance, maintaining both speed and semantic accuracy
Data Source
AI summary
The present disclosure provides a corpus cleaning method and a corpus entry system. The method includes: obtaining an input utterance; generating a predicted value of an information amount of each word in the input utterance according to the context of the input utterance using a pre-trained general model; and determining redundant words according to the predicted value of the information amount of each word, and determining whether to remove the redundant words from the input utterance. In such a manner, the objectivity and accuracy of corpus cleaning can be improved.