Data Cleansing via Intersection Sets and Concept Replacement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data cleansing methods are inadequate in accurately distinguishing between noise and important concepts in natural language processing, particularly in multi-domain contexts where words can have different meanings based on domain-specific contexts.
Innovation Solution
A system and method that categorize data records, identify concepts, group them into category lists, determine intersection sets of common concepts between category lists, and replace common concepts with category names based on occurrence rates to improve classification accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional data cleansing methods are used, then the process is simple and fast, but the accuracy of distinguishing noise from important concepts deteriorates
Solution Approach 1:
The patent segments the data cleansing process into multiple stages: initial categorization, concept identification, grouping into category lists, determining intersection sets, and replacing common concepts. This segmentation allows the system to handle complexity systematically while improving accuracy at each stage.
Solution Approach 2:
The patent performs preliminary categorization of data records into domains before concept identification. This preliminary action establishes the contextual framework needed to accurately distinguish between noise and important concepts, resolving the contradiction by preparing the data structure in advance.
2Measurement precision
If domain-specific context analysis is implemented, then classification accuracy improves, but processing time increases
Solution Approach 1:
The system performs preliminary domain categorization of data records before detailed concept analysis. By establishing the domain context upfront, the system reduces the computational burden of subsequent concept identification and intersection set determination, thereby improving classification accuracy without excessive time loss.
Solution Approach 2:
The patent extracts and replaces common concepts from intersection sets with category names. This extraction process eliminates redundant analysis of frequently occurring concepts, reducing processing time while maintaining classification accuracy through the use of pre-identified category labels.
3Measurement precision
If intersection sets and common concepts are identified and replaced, then data classification accuracy improves, but the complexity of the system increases
Solution Approach 1:
The patent segments the complex task of improving classification accuracy into manageable components: identifying intersection sets between category lists, determining common concepts within those sets, and replacing them with category names. This segmentation makes the system's complexity tractable while achieving the goal of improved accuracy.
Solution Approach 2:
The system creates category lists as copies or representations of domain-specific concept groups. By working with these copied structures rather than the original complex data, the system reduces computational complexity while maintaining the ability to accurately identify and replace common concepts.
Data Source
AI summary
System and method for data cleansing are disclosed. The method comprises receiving one or more data records pre-categorized into one or more categories. Identifying at least one concept associated with one or more data records, and grouping, the at least one concept associated with the one or more data records into a plurality of category lists based on the predefined category associated with each of the one or more data records. Determining, one or more intersection sets based on a comparison between each pair of the plurality of category lists, wherein each intersection set comprises a set of one or more common concepts associated with a pair of category lists. The method comprises replacing each of at least one common concept of the set of one or more common concepts associated with each intersection set by a category name based on an occurrence rate of the common concepts.