Classification Model Training Using Association Rule Lift Aggregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data mining techniques face inefficiencies in processing large datasets, particularly in manufacturing and IT systems, due to slow processing speeds, high memory consumption, and the need for item hierarchies, which can lead to poor classification model quality when hierarchies are flawed or absent.
Innovation Solution
A method for training a classification model that uses association rules to link entity classifications with items, determining applicable rules, and aggregating lift values to predict classifications without relying on item hierarchies, allowing for efficient processing and improved model quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If breadth-first search is used to mine association rules, then all candidate patterns can be evaluated, but processing speed becomes slow due to multiple scans on large datasets
Solution Approach 1:
The patent pre-processes the dataset to compute item frequencies and creates an inverted index structure before the actual association rule mining. This preliminary organization of data allows the algorithm to quickly retrieve and evaluate candidate patterns without repeatedly scanning the entire dataset, thus maintaining completeness while improving processing speed
Solution Approach 2:
The patent introduces an inverted index as an intermediary data structure that maps items to the transactions containing them. This intermediary allows efficient access to relevant data subsets during pattern evaluation, avoiding the need to scan all transactions for each candidate pattern, thereby resolving the contradiction between thorough evaluation and processing speed
2Quantity of substance
If depth-first search is used to mine association rules, then memory usage is reduced, but the algorithm runs out of memory for very large datasets
Solution Approach 1:
The patent segments the association rule mining process into distinct phases: generating candidate patterns, evaluating candidates against the inverted index, and aggregating results. This segmentation allows the algorithm to process data in manageable chunks rather than loading all candidate patterns into memory simultaneously, enabling handling of very large datasets while controlling memory usage
Solution Approach 2:
The patent transitions from the traditional item hierarchy dimension to a new dimension based on the inverted index structure. Instead of organizing candidates by hierarchical levels, the algorithm organizes evaluation around item-based indexing, creating a new dimensional approach that scales better with dataset size and reduces memory requirements
3Ease of operation
If item hierarchy is determined for data mining, then the algorithm can structure its search, but the determined hierarchy may be flawed and discredit the results
Solution Approach 1:
The patent extracts and removes the dependency on item hierarchy from the association rule mining process. By using the inverted index and frequent itemset-based candidate generation, the algorithm achieves structured search capability without requiring a pre-defined or determined item hierarchy, thereby eliminating the source of potential errors that could discredit results
4Quantity of substance
If standard classification algorithms are used with multiple rows per entity, then all available data can be utilized, but the input table format requirement causes processing inefficiencies
Solution Approach 1:
The patent performs preliminary aggregation of multiple rows per entity into a single consolidated record during the candidate pattern generation phase. By pre-computing item frequencies and candidate patterns from all rows of an entity before evaluation, the algorithm utilizes all available historical data while maintaining processing efficiency in the subsequent evaluation phases
Data Source
AI summary
A classification model is trained and used for detecting patterns in input data. The training of the model includes retrieving a set of previously recorded input data containing a plurality of items associated with a plurality of entities and adding to each entity a known classification. Furthermore, training the model includes determining rules from the set of previously recorded input data and the known classification by associating the classification of each entity with the respective items of said entity. The training of the model further includes determining a set of rules which are applicable, aggregating the lift values of the rules determined for said entity, and predicting a classification based on the aggregated association values for each entity. The resulting aggregated lift value together with the respective entity and classification are used as input for a standard classification algorithm, where the result is a classification model.