Parallel Association Analysis Using Projection Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional association analysis methods face low execution efficiency due to the excessive data volume transmitted during data summarization in parallel association analysis, particularly in sequential-pattern association analysis, where candidate frequent item sets and their support counts require large-scale data merging.
Innovation Solution
The method involves dividing an original database into projection databases where one does not contribute to the support count of another, allowing nodes to perform sequential-pattern association analysis separately to obtain local frequent item sets and support counts, which are then combined to obtain global results, thereby reducing data transmission overheads and improving efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is fragmented and separate association analysis is performed on each piece at each node, then parallel processing capability is improved, but the data volume of candidate frequent item sets generated by nodes becomes exponentially larger than the original fragmented data, resulting in excessive data transmission during merging
Solution Approach 1:
The original database is divided into multiple projection databases, each assigned to different nodes for parallel processing. Each projection database is created by projecting the original database onto a specific subset of items, ensuring that the segmentation maintains the ability to compute support counts while reducing the exponential growth of candidate frequent item sets at each node.
Solution Approach 2:
The invention extracts and removes redundant candidate frequent item sets that would otherwise be generated and transmitted during the merging phase. By using projection databases with carefully selected item subsets, the method extracts only the necessary data for each node's local computation, eliminating the need to handle and transmit the full exponential set of candidates across the network.
2Measurement precision
If candidate frequent item sets and support counts from multiple nodes are merged to obtain global support counts, then global frequent item sets can be obtained, but an excessively large volume of data is transmitted during the merge operation, resulting in low execution efficiency
Solution Approach 1:
The computation of global support counts is segmented into local support count computations performed independently at each node on its assigned projection database. This segmentation allows each node to compute support counts for its local candidate frequent item sets without needing to transmit all candidate data, as the projection structure ensures that each item's support count can be computed locally and then aggregated.
Solution Approach 2:
The invention performs preliminary computation of support counts at each node before the merging phase. By computing local support counts in advance on projection databases, the method prepares the necessary aggregated data for global frequent item set determination, eliminating the need to transmit and process large volumes of raw candidate frequent item sets during the merge operation.
Data Source
AI summary
There is provided an association analysis method and apparatus. An original database is divided into projection databases, each not contributing to a support count of a frequent item set of another. The projection databases are used for sequential-pattern association analysis performed respectively by nodes corresponding to the projection databases. Local frequent item sets and corresponding support counts obtained by the nodes are combined. Since an established projection database does not contribute a support count of a frequent item set of another projection database, different nodes can perform association mining, including pruning, on different projection databases respectively.