Association Rule Mining via Local-Global Conditional FP Trees
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current association rule mining techniques, such as parallel frequent pattern growth, face issues with high memory usage, processing time, and data redundancy, leading to inefficient resource utilization and increased risk of memory overflow, especially when dealing with large datasets.
Innovation Solution
The method generates local conditional frequency pattern trees in processing nodes, which are then aggregated to form global conditional frequency pattern trees distributed across nodes, allowing for efficient pattern generation and rule definition without the need for storing all trees in memory, thereby reducing memory usage and processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If parallel frequent pattern growth technique is used to perform frequent item discovery, then the number of patterns that can be identified increases, but memory usage becomes excessively large and processing time increases
Solution Approach 1:
The patent divides the dataset into multiple partitions and processes each partition independently to generate local frequent pattern trees. This segmentation allows parallel processing across multiple processing nodes while keeping memory usage at each node manageable, as each node only needs to store and process one partition at a time rather than the entire dataset
Solution Approach 2:
The patent introduces a new dimension of processing by creating local frequent pattern trees for each partition and then combining them through a global aggregation process. This multi-dimensional approach (local trees per partition + global combination) enables scalable pattern discovery without linearly increasing memory requirements
2Productivity
If parallel frequent pattern growth technique is used to perform frequent item discovery, then the number of patterns that can be identified increases, but processing time becomes extremely long
Solution Approach 1:
By dividing the dataset into partitions and processing them in parallel across multiple nodes, the overall processing time is reduced compared to sequential processing of the entire dataset, while maintaining the ability to discover all frequent patterns
Solution Approach 2:
The patent performs preliminary processing by generating local frequent pattern trees from each partition before combining them globally. This preliminary action at the local level enables more efficient global aggregation, reducing the overall processing time compared to processing the entire dataset as a single unit
3Adaptability or versatility
If multiple conditional FP trees are generated in parallel item discovery, then frequent item set mining can be performed, but redundant data is present and updating these trees is time-consuming
Solution Approach 1:
The patent extracts only the necessary information from each local frequent pattern tree that is needed for global pattern discovery. By taking out and keeping only the essential frequent patterns and their support counts, the method avoids storing and processing redundant data while maintaining the ability to perform comprehensive frequent item set mining
4Adaptability or versatility
If multiple conditional FP trees are generated in parallel item discovery, then frequent item set mining can be performed, but memory used becomes very large
Solution Approach 1:
The patent segments the memory requirements by creating separate local frequent pattern trees for each data partition rather than creating all possible conditional trees in memory simultaneously. This segmentation of memory usage allows the system to handle large datasets without requiring excessive memory resources
Solution Approach 2:
The patent discards local frequent pattern trees after their information has been aggregated into the global structure, recovering memory space for processing the next partition. This approach enables the system to maintain only necessary data in memory at any given time, significantly reducing peak memory consumption
Data Source
AI summary
Identifying relations between items in transactions. Local conditional frequency pattern trees are generated in processing nodes in a computer system. The nodes in the local conditional frequency pattern trees represent items and a frequency of the items in the transactions. Global conditional frequency pattern trees are generated in the processing nodes. Each processing node in the processing nodes generates a global conditional frequency pattern tree using a set of local conditional frequency pattern trees in the processing node. The global conditional frequency pattern trees generated by the processing nodes are distributed such that each processing node in the processing nodes has the global conditional frequency pattern trees generated by other processing nodes. Patterns are generated by the computer system. A set of rules is generated using the patterns and the global conditional frequency pattern trees in the processing nodes, wherein the set of rules define the relations between the items.