Character-Based Text Term Scoring for Automated Term Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text processing systems struggle to effectively determine the specificity of text terms, which is crucial for information retrieval and knowledge discovery, due to the vast and diverse nature of modern data sources like research papers and news reports.
Innovation Solution
A computer-implemented method using a character-based regression model to predict the specificity of text terms by training on a dataset and masking terms below a threshold, enabling efficient and accurate specificity scoring for downstream applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional text processing methods are used to determine specificity of text terms, then measurement precision may be maintained, but computational efficiency and resource requirements deteriorate
Solution Approach 1:
The patent transforms the specificity determination problem into a parameter prediction task by training a character-based regression model to predict specificity scores directly from text term characters. This changes the approach from complex traditional text processing to a streamlined parameter prediction task, achieving both computational efficiency and measurement precision through the learned parameter mapping.
Solution Approach 2:
The patent replaces traditional mechanical text processing systems with a machine learning-based regression model. Instead of using rule-based or statistical text processing methods, the system employs a neural network model that learns patterns from training data, substituting the mechanical processing approach with an intelligent prediction system that achieves better efficiency and accuracy.
2Measurement precision
If comprehensive text processing is performed on all terms, then measurement precision is improved, but device complexity and resource requirements increase
Solution Approach 1:
The patent extracts only the essential character-level features from text terms for the regression model, rather than performing comprehensive text processing. By taking out and using only the necessary character information to predict specificity scores, the system reduces device complexity and resource requirements while maintaining measurement precision through the learned character-based prediction mechanism.
3Quantity of substance
If low specificity terms are included in training data, then data quantity is maintained, but training data quality and model performance deteriorate
Solution Approach 1:
The patent applies local quality filtering by masking specific low-specificity text terms within the training data while preserving the overall data structure. Instead of uniformly reducing data quality, the system selectively masks only those terms that would degrade model performance, maintaining high-quality training data regions while removing problematic local regions, thus preserving both data quantity and overall reliability.
Data Source
AI summary
A computer implemented method, a computer program product and a computer system and are provided to enrich downstream learning tasks. A processor stores selected text terms from a corpus of text. A processor determines an initial set of specificity scores for the selected text terms to produce a set of training samples, where each of the training samples comprise a selected text term and an initial specificity score for the selected text term. A processor trains a character-based regression model with the set of training samples. A processor retrieves an Automated Term Extraction (ATE) training data set. A processor determines specificity scores for text terms included in the ATE training data set. A processor, responsive to respective specificity score for a text term in the ATE training data set being below a threshold value, masks the text term from being used in the ATE training data set.