Statistical Subject Identification via N-gram Weighting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing approaches to subject identification in Natural Language Processing require large amounts of annotated data for training, leading to time-consuming efforts and inconsistent results due to training data bias, especially when different models produce varying outcomes for the same input text data.
Innovation Solution
A processor-implemented method for statistical subject identification that processes various input data types (text, images, audios, videos) by converting them into text strings, calculating n-gram weightages, applying Box-Cox transformation for confidence values, and mapping significant nodes to domains and subject areas without relying on pre-trained models, thus minimizing bias.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a pre-trained model is used to identify subject of unseen data, then the model can solve problems similar to training data, but a large amount of annotated data is required for training which consumes time and effort
Solution Approach 1:
The patent applies preliminary action by pre-processing input text data into n-grams and calculating their frequencies before subject identification. This preparation step transforms raw text into a structured format (n-gram frequency distributions) that can be directly analyzed using statistical methods, eliminating the need for time-consuming model training while maintaining identification accuracy
Solution Approach 2:
The patent replaces the mechanical training process of traditional machine learning models with a statistical methodology based on n-gram frequency analysis. Instead of training neural networks or other complex models on annotated data, the system uses mathematical statistics to identify subjects, thereby eliminating training time while preserving reliability
2Adaptability or versatility
If different models are created with different sets of training data, then various subject identification approaches can be explored, but varying results are obtained for the same input text data
Solution Approach 1:
The patent applies parameter changes by using statistical parameters (n-gram frequencies, confidence intervals, significance thresholds) instead of fixed model parameters learned during training. This statistical approach with configurable parameters provides both adaptability to different input types and consistency in results, as the same statistical methodology yields reproducible outcomes regardless of input variations
3Reliability
If a pre-trained model is created with specific training data, then the model can be trained to solve specific problems, but bias is introduced due to the choice of data which produces inconsistent results
Solution Approach 1:
The patent extracts the essential information from training data by computing n-gram frequencies directly from the input text itself, rather than relying on external annotated training datasets. This extraction approach eliminates training data bias while maintaining identification accuracy, as the statistical patterns are derived from the input data's own structure
Solution Approach 2:
The patent applies self-service by having the input text data serve its own training purpose. The n-grams are extracted and analyzed from the input text itself, making the data self-descriptive and eliminating dependency on external annotated datasets. This reduces complexity by using the input data's inherent structure for identification
Data Source
AI summary
Embodiments provide a system and method for statistical subject identification. The system takes texts, videos, audios, and images as input for which subject needs to be identified. The system pre-process input data and generates n-grams and pre-processed text strings by removing stopwords, punctuations, selective POS tags and lemmatization. Frequency distribution of n-grams are computed, and weightage of n-grams is assigned. For each n-gram, sum of weights across all text strings is computed and a maximum weightage is identified. The computed value as a result of taking a ratio of two, is assigned to each of the n-grams. Values computed for the n-grams have a non-normal distribution, when observed statistically. Thus, the n-gram values are transformed to confidence value following a normal distribution. The system maps the n-gram domains using a domain lexicon. Finally, these domains are aggregated and converged for subject identification based on a pre-annotated mapping dictionary.