Structured Rule Clustering for Faster Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The inefficiencies in computer operations due to the large number of redundant and overlapping rules used in data processing, particularly in big data scenarios, lead to increased processing times and resource consumption.
Innovation Solution
A method involving filtering and selecting one rule from a plurality of rules, vectorizing its expression with different dimensions, and building a cluster model based on tree-similarity related distances, utilizing machine learning to optimize structured rules and reduce redundancy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a large number of rules are used in data processing to ensure comprehensive data analysis, then data extraction capability is improved, but processing time and resource consumption increase
Solution Approach 1:
The patent extracts and removes redundant rules from the rule set through clustering analysis. By identifying rules that are semantically similar or duplicate each other's function, the system eliminates unnecessary rules while preserving the essential data extraction capabilities, thus resolving the contradiction between comprehensive analysis and processing efficiency
Solution Approach 2:
The patent merges redundant rules into clustered groups based on similarity metrics. Rules that cover the same or overlapping data patterns are combined into single representative rules or rule clusters, reducing the total number of rules processed while maintaining the same data extraction effectiveness
2Reliability
If thousands of rules are applied to big data to ensure thorough analysis, then data coverage is improved, but computational resource consumption increases
Solution Approach 1:
The system extracts only the essential, non-redundant rules from the thousands of rules in the dataset. Through vector representation and clustering, it identifies and removes duplicate rules, keeping only the necessary subset that provides complete data coverage with minimal computational overhead
Solution Approach 2:
The patent changes the parameter of rule representation by converting rules into vector forms and applying clustering algorithms. This transformation enables efficient identification and removal of redundant rules, significantly reducing computational resource consumption while preserving data coverage through the optimized rule set
3Reliability
If redundant rules are maintained in the rule set to handle edge cases, then rule robustness is improved, but processing overhead increases
Solution Approach 1:
The patent merges rules that handle similar edge cases into unified rule clusters. By analyzing semantic similarity and functional overlap, it combines redundant edge case handlers into single robust rules that cover multiple scenarios, eliminating processing overhead from duplicate edge case handling while maintaining rule robustness
Data Source
AI summary
A computer-implemented method and device to optimize structured rules in data processing. The method includes filtering and selecting one rule from a plurality of rules. An expression of the one rule is vectorized with different dimensions. A cluster model of rules is built from at least some of the plurality of rules having a vector distance that is replaced by a specific vector distance. The building of the cluster model of rules is based on identifying a tree-similarity related distance of some of the plurality of rules from the selected one rule.