Weighted N-Gram Preprocessing for Redundant NLP Training Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing natural language processing (NLP) models suffer from reduced accuracy due to insufficient preprocessing of training data, which is often performed ad-hoc and incomplete, leading to redundancies that hinder performance.
Innovation Solution
A systematic and comprehensive preprocessing method is applied to training data, involving tokenization, n-gram tuple determination, weight assignment, and threshold-based classification to identify and remove redundancies, enhancing the quality of training data for NLP models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If ad-hoc preprocessing is performed on training data, then the preprocessing process is simple and quick, but NLP model accuracy suffers due to insufficient redundancy elimination
Solution Approach 1:
The patent applies preliminary action by performing systematic preprocessing of training data before NLP model training. The method involves tokenizing text, generating n-grams, calculating redundancy scores, and filtering training examples in advance to eliminate redundancies before the model training process begins, thereby improving model accuracy without adding complexity during training
Solution Approach 2:
The patent applies segmentation by breaking down the preprocessing task into distinct stages: tokenization of text into individual tokens, generation of n-gram tuples from tokens, calculation of redundancy scores for each training example, and filtering based on threshold criteria. This segmented approach makes the complex preprocessing systematic and manageable
2Reliability
If comprehensive preprocessing is applied to eliminate all redundancies, then NLP model accuracy improves, but processing time and computational resources increase
Solution Approach 1:
The patent applies parameter changes by using configurable parameters such as n-gram size (e.g., 2-grams, 3-grams), redundancy threshold values, and minimum example counts. These parameters can be adjusted to balance between comprehensive redundancy elimination and processing efficiency, allowing optimization of preprocessing time based on specific requirements
Solution Approach 2:
The patent applies partial action by filtering training examples based on redundancy thresholds rather than eliminating all possible redundancies. The method identifies and removes training examples that exceed specified redundancy criteria, performing a targeted subset of the maximum possible preprocessing to achieve sufficient accuracy improvement without excessive processing time
Data Source
AI summary
An embodiment may involve: obtaining textual content including a plurality of token strings, wherein each of the plurality of token strings includes one or more tokens; determining, for the plurality of token strings, respectively corresponding sets of n-gram tuples; assigning respective weights to the plurality of token strings, wherein, for each of the plurality of token strings, the assignment is based on the respectively corresponding set of n-gram tuples; identifying a subset of the plurality of token strings, wherein each of the subset of the plurality of token strings is characterized by a respective weight that exceeds a predetermined threshold weight; and storing sets of n-gram tuples respectively corresponding to the subset of the plurality of token strings.