Distributed Pattern Mining Algorithm for Large Datasets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data mining techniques struggle with efficiently extracting reliable and relevant patterns from large datasets, especially when dealing with rough data sets and continuous attributes, as they often fail to handle non-linear relationships and require significant computational resources, leading to errors and inefficiencies.
Innovation Solution
A distributed algorithm that preprocesses data by computing class distribution and discretizing continuous attributes, then generates and refines patterns across multiple computing buckets, evaluating their reliability, significance, and relevance through statistical tests, allowing for efficient pattern extraction and reduction of the search space without relying on heuristics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing data mining techniques use greedy algorithms such as gradient descent to identify patterns, then the techniques can optimize for a given accuracy, but they cause errors when working with rough data sets and continuous attributes due to computational intensity and inability to handle non-linear relationships
Solution Approach 1:
The patent segments the continuous attribute space into discrete intervals using equal-width binning, transforming the continuous data into discrete form that can be processed by the pattern mining algorithm. This segmentation allows the system to handle continuous attributes without requiring complex gradient-based optimization, thereby improving reliability while reducing computational complexity.
Solution Approach 2:
The patent changes the parameter representation from continuous values to discrete interval identifiers. By transforming continuous attributes into discrete bins and using frequency-based pattern evaluation instead of gradient descent, the system achieves better reliability on rough datasets while reducing computational complexity to linear or near-linear time.
2Ease of operation
If decision trees break down the range of values into intervals and prioritize them using information gain, then the technique is easy to understand, but it fails to address which attributes are important and the select values that are important in different regions of the data
Solution Approach 1:
The patent generates all possible attribute-value patterns rather than selecting only the top splits, then evaluates them using frequency and statistical significance. This exhaustive approach with subsequent filtering preserves information about attribute importance across different data regions while maintaining interpretability through the frequency-based ranking of patterns.
3Reliability
If the system evaluates each pattern for reliability, refinability and relevance through statistical tests, then the system extracts reliable and relevant patterns, but the computational complexity increases significantly
Solution Approach 1:
The patent performs preliminary frequency counting and pattern generation in a first pass through the data, storing patterns that meet minimum frequency thresholds. In a second pass, it performs statistical significance testing only on these pre-filtered patterns. This preliminary action reduces the number of patterns requiring expensive statistical tests, thereby maintaining reliability while improving productivity.
Data Source
AI summary
System pre-processes and computes class distribution of decision attribute and statistics for discretization of continuous attributes through use of compute buckets. System computes the variability of each of the attributes and considers only the non-zero variability attributes. System computes the discernibility strength of each attribute. The software system generates size 1 patterns using compute bucket and calculates if each pattern of size 1 is a reliable pattern for any class. The system calculates if reliable pattern of size 1 is a significant pattern for any class. The system generates size k patterns from size k−1 patterns checking for significance of size k patterns and refinability. The system readjusts pattern statistics for only significant patterns for size k−1 patterns. The system computes a cumulative coverage of the sorted relevant patterns of up to size k by finding out the union of records of that particular class.