Data Cleansing via Overlap Ratio Normalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data classification systems face inaccuracies due to common words that appear across various categories, making it difficult to determine whether a word belongs to noise or is important, leading to less precise classification.
Innovation Solution
A method and system for data cleansing that groups words into pre-defined categories, calculates overlap ratios, computes normalized overlap ratios, and removes common words based on frequency of occurrence to improve data classification accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional data classification approaches are used, then the classification process is simple, but the classification accuracy deteriorates due to common words appearing across various categories
Solution Approach 1:
The patent segments the data classification process into distinct stages: initial classification, overlap detection, and iterative refinement. By dividing the classification task into manageable segments with specific focus on identifying and handling overlapping categories, the system achieves higher accuracy without overwhelming complexity
Solution Approach 2:
The patent implements a feedback mechanism where classification results are continuously evaluated, and overlap ratios are calculated to inform subsequent classification decisions. This iterative feedback loop allows the system to learn from previous classifications and improve accuracy by adjusting for common words across categories
2Measurement precision
If data cleansing is performed to remove common words, then classification precision improves, but the processing time and computational resources increase
Solution Approach 1:
The patent performs preliminary action by pre-identifying and flagging common words that appear across multiple categories before the actual classification process. This preliminary preparation allows the classification algorithm to efficiently handle overlapping terms without extensive processing during the main classification phase
Solution Approach 2:
The patent changes parameters by dynamically adjusting the threshold for what constitutes a 'common word' based on frequency analysis and category overlap ratios. This parameter adjustment allows the system to adapt to different data sets and maintain precision while optimizing processing efficiency
3Measurement precision
If overlap ratios are calculated between all category pairs, then classification accuracy improves, but the computational complexity increases
Solution Approach 1:
The patent extracts only the necessary overlap information by identifying and focusing on words that appear in multiple categories. Rather than calculating all possible category relationships, the system extracts and processes only the relevant overlaps that actually impact classification decisions, reducing computational burden while maintaining accuracy
Data Source
AI summary
A system and method for data cleansing are disclosed. The method includes receiving by a data classifier, a data corpus comprising one or more words. The method further includes grouping the one or more words into a plurality of pre-defined categories and determining an overlap ratio between each pair of the plurality of pre-defined categories based on a comparison between each of the plurality of pre-defined categories. A normalized overlap ratio is computed for each of the plurality of pre-defined categories based on a combination of each overlap ratio. At least one of the plurality of pre-defined categories is determined based on a comparison between each of the normalized overlap ratio and an ideal overlap ratio. Finally, one or more common words from the selected at least one of the plurality of pre-defined categories is removed based on a frequency of occurrence of the one or more common words.