Prefix Tree Mining for Population-Normalized Patterns
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data mining techniques, such as the Apriori and FP-growth algorithms, are computationally expensive and inefficient when dealing with large datasets, particularly in identifying frequent patterns in high-dimensional sparse feature spaces, as they fail to account for the overall popularity of patterns in the general population.
Innovation Solution
A method that involves generating a list of frequent items from a subpopulation and building a prefix tree using a Monoid algebraic structure to accumulate counts of both subpopulation and general population records, allowing for the extraction of population-normalized frequent patterns in a single pass through the dataset, reducing complexity and execution time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If Apriori algorithm is used to mine frequent patterns, then frequent patterns can be discovered, but computational cost and memory consumption increase significantly
Solution Approach 1:
The patent segments the frequent pattern mining process into two distinct phases: (1) generating candidate patterns using Apriori's downward closure property, and (2) filtering and counting only those candidates that appear in the target subpopulation. This segmentation avoids the computational waste of counting all candidate patterns in the entire database, thereby resolving the contradiction between reliable pattern discovery and computational efficiency.
Solution Approach 2:
The patent extracts and isolates the counting operation to only apply to candidate patterns that are relevant to the target subpopulation. By taking out the unnecessary counting operations for patterns not present in the subpopulation, the algorithm maintains reliability in discovering frequent patterns while significantly improving computational efficiency by avoiding redundant calculations.
2Productivity
If FP-growth algorithm is used to mine frequent patterns, then execution time is reduced, but ability to find patterns in specific groupings is lost
Solution Approach 1:
The patent performs preliminary action by first generating the candidate pattern set using Apriori's efficient candidate generation, then immediately filters this candidate set to retain only patterns present in the target subpopulation before performing counting. This preliminary filtering action ensures that the subsequent counting phase only processes relevant patterns, maintaining both execution efficiency and subpopulation-specific pattern discovery accuracy.
Solution Approach 2:
The patent introduces an intermediary filtering step between candidate generation and pattern counting. This intermediary process acts as a mediator that selects which candidates deserve to be counted based on their presence in the target subpopulation, thereby bridging the gap between the efficiency of FP-growth and the subpopulation-specific accuracy requirement.
3Measurement precision
If traditional frequent pattern mining is performed without population normalization, then pattern frequency can be calculated, but overall popularity context is lost
Solution Approach 1:
The patent implements feedback by calculating the proportion of each frequent pattern's occurrence in the target subpopulation relative to its occurrence in the entire population. This feedback mechanism provides context about whether a pattern is over-represented or under-represented in the subpopulation, preventing loss of population context information while maintaining precise frequency measurement through the normalized proportion calculation.
Data Source
AI summary
Disclosed are systems and methods for data mining a plurality of records to identify one or more patterns. A list of frequent items is generated using the records of a certain subpopulation in a dataset of the records. By scanning through the dataset, a prefix tree is generated based on the list of frequent items. Each node in the prefix tree includes an accumulator which maintains separate counts of records from the subpopulation matching the respective node and of records from the plurality of records matching the respective node. One or more population-normalized frequent patterns associated with the plurality of records are extracted based on a traversal of the prefix tree.