N-gram Score Sorting for Language Processing Training Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Obtaining training data for language processing engines, such as POS tagging engines, is difficult and expensive due to the vast amount of potential inputs, making it impractical to manually tag all data, especially in high-traffic social media platforms where manual tagging is not feasible.
Innovation Solution
A data mining system that identifies high-coverage training data by assigning an n-gram score to potential snippets, sorting them based on this score, and selecting the top-scoring snippets for training, thereby improving language coverage without requiring manual tagging of all data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If manual tagging of all potential inputs is performed to obtain training data, then the quality and coverage of training data is improved, but the cost and time required becomes prohibitively high
Solution Approach 1:
The patent applies partial action by selecting and processing only a subset of potential training data rather than manually tagging all possible inputs. The system identifies and processes snippets that meet specific criteria (recency, source reliability, language characteristics) to create training data that achieves sufficient coverage without requiring exhaustive manual tagging of all available data
Solution Approach 2:
The system performs preliminary filtering and selection of snippets based on predefined criteria before manual tagging is attempted. By pre-identifying high-value candidates based on source, recency, and linguistic properties, the system prepares the data in advance to maximize the efficiency of subsequent manual tagging efforts
2Manufacturing precision
If manual tagging of all potential inputs is performed to obtain training data, then the quality and coverage of training data is improved, but the cost becomes prohibitively high
Solution Approach 1:
The system processes only a carefully selected subset of potential training data rather than attempting to manually tag all possible inputs. By applying filtering criteria that identify the most valuable snippets based on source reliability, recency, and language characteristics, the system achieves cost-effective training data creation with sufficient coverage
Solution Approach 2:
The system automatically performs preliminary filtering, selection, and preparation of training data candidates without requiring human intervention for every snippet. The automated identification of high-value candidates based on predefined criteria reduces the overall cost by eliminating the need for expensive manual tagging of low-value data
3Productivity
If a small subset of data is tagged manually to reduce cost and time, then resource consumption is reduced, but the language coverage and effectiveness of the training data decreases
Solution Approach 1:
The system applies different selection criteria to identify snippets with high linguistic value. By focusing on specific sources, recency thresholds, and language characteristics, the system ensures that the subset of manually tagged data represents diverse and linguistically rich content that maximizes language coverage relative to the amount of data processed
Solution Approach 2:
The system adjusts selection parameters such as source priority weights, recency thresholds, and language-specific criteria to optimize the balance between the quantity of data processed and the quality of language coverage achieved. By tuning these parameters, the system can adapt to different resource constraints while maintaining effective training data coverage
Data Source
AI summary
Technology is disclosed that improves language coverage by selecting sentences to be used as training data for a language processing engine. The technology accomplishes the selection of a number of sentences by obtaining a group of sentences, computing a score for each sentence, sorting the sentences based on their scores, and selecting a number of sentences with the highest scores. The scores can be computed by dividing a sum of frequency values of unseen words (or n-grams) in the sentence by a length of the sentence. The frequency values can be based on posts in one or more particular domains, such as the public domain, the private domain, or other specialized domains.