Training Data Concept Filtering for Classification Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data cleansing methods fail to effectively remove relevant but unusable noise while retaining usable noise, which hampers the accuracy of data classification.
Innovation Solution
A method and system that extract concepts from a training data set, compute their frequency and percentage contribution, remove concepts with low frequency or equal contribution, and append category names based on normalized frequency to improve data classification accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If existing data cleansing methods are used to remove noise, then data processing is simplified, but data classification accuracy deteriorates because usable noise is removed along with relevant but unusable noise
Solution Approach 1:
The patent segments noise into three distinct categories: relevant but unusable noise (removed via frequency thresholding), usable noise (retained via contribution analysis), and irrelevant noise (removed). This segmentation allows selective processing that maintains classification accuracy while simplifying data processing.
Solution Approach 2:
The patent applies different processing rules to different types of noise based on their local characteristics. Relevant but unusable noise is removed using frequency thresholding, while usable noise is retained through contribution analysis. This localized quality approach ensures each noise type is handled appropriately, improving classification accuracy without excessive complexity.
2Reliability
If all noise is removed from training data, then data quality improves, but information loss occurs because usable noise contains category-specific information
Solution Approach 1:
The patent changes the parameter of noise evaluation from binary (noise vs. signal) to a continuous assessment using frequency of occurrence and percentage contribution metrics. This allows the system to identify and retain usable noise that contains category-specific information while removing noise that degrades classification performance.
Solution Approach 2:
The patent uses feedback mechanisms by computing the percentage contribution of each concept to each category and using this information to determine whether to retain or remove the concept. This feedback loop ensures that usable noise containing category-specific information is preserved while maintaining high data quality.
3Measurement precision
If frequency thresholding is applied to remove low-frequency concepts, then relevant but unusable noise is removed, but some usable noise may be incorrectly removed
Solution Approach 1:
The patent performs preliminary frequency thresholding to remove obvious irrelevant noise, then follows up with contribution analysis to recover any usable noise that was incorrectly removed. This two-stage preliminary action ensures thorough noise removal while preserving category-specific information.
Solution Approach 2:
The patent introduces contribution analysis as an intermediary step between frequency thresholding and final data selection. This intermediary mechanism acts as a filter that rescues usable noise incorrectly identified as irrelevant by frequency thresholding alone, balancing noise removal accuracy with information retention.
4Measurement precision
If contribution analysis is performed on all concepts, then usable noise is retained, but computational complexity increases
Solution Approach 1:
The patent applies contribution analysis selectively rather than to all concepts. Frequency thresholding first removes obvious irrelevant noise, and contribution analysis is then applied only to assess the remaining concepts. This partial action approach maintains noise classification accuracy while reducing unnecessary computational complexity.
Data Source
AI summary
In one embodiment, a method for transforming training data to improve data classification is disclosed. The method comprises extracting concepts from a training data set. The method comprises computing frequency of occurrence of each concept in each category and removing concepts from the data records when the frequency of occurrence of a concept in a category is less than a threshold frequency value. Further, the method comprises computing a percentage contribution of each concept of remaining concepts in each category upon removing the concepts and eliminating concepts, from the remaining concepts, contributing equally to each category based on the percentage contribution of each concept to provide a reformed training data set. Further, the method comprises appending a category name to a corresponding data record in the reformed training data set based on a normalized frequency of occurrence of the concept in a category to improve data classification.