Derived Field Clustering for Faster Ensemble ML Predictions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data analytics pipelines, particularly ensemble machine learning models, face significant delays due to the slow processing of numerous derived fields, which hinders real-time prediction capabilities, especially in scenarios requiring immediate detection of fraudulent activities.
Innovation Solution
A system and method for efficient transformation prediction in data analytics pipelines, involving clustering derived fields based on Derived Field Importance Weight for Field (DFIW4F) and Derived Field Importance Weight for Model (DFIW4M), prioritizing and parallel processing to optimize the run time of base machine learning models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If numerous derived fields are processed in data analytics pipelines, then the comprehensiveness of machine learning model input is improved, but the processing time increases significantly
Solution Approach 1:
The patent segments the processing of derived fields by organizing them into a hierarchical structure with root derived fields and child derived fields. This segmentation allows the system to process only the necessary subset of derived fields (those in the shortest path from root to target) rather than all possible derived fields, thereby reducing processing time while maintaining the completeness of feature engineering for the specific prediction task.
Solution Approach 2:
The patent implements preliminary action by pre-computing and storing the hierarchical relationships between derived fields, including the identification of root derived fields and the calculation of shortest paths. This pre-processing creates a derived field hierarchy that can be quickly queried during prediction, eliminating the need to re-analyze all derived field relationships at prediction time and significantly reducing latency.
2Measurement precision
If all base machine learning models are run with all derived fields, then the accuracy of predictions is improved, but the computational complexity and processing time increase
Solution Approach 1:
The patent extracts only the essential derived fields needed for accurate prediction by identifying the shortest path from root derived fields to target derived fields. This extraction principle removes unnecessary derived fields and base models from the processing pipeline, reducing computational complexity while retaining the accuracy-contributing components. The system takes out only what is necessary rather than processing everything.
Solution Approach 2:
The patent applies local quality by treating different derived fields differently based on their importance and relationship to the prediction target. Rather than uniformly processing all derived fields with the same computational resources, the system identifies and prioritizes the local subset of derived fields (those in the shortest path) that have the most significant impact on prediction accuracy, allocating processing resources accordingly.
3Speed
If real-time fraud detection is implemented, then the timeliness of suspicious activity detection is improved, but the processing speed requirement increases the difficulty of handling large data volumes
Solution Approach 1:
The patent segments the large volume of data and derived fields into a focused subset by identifying the shortest path through the derived field hierarchy. This segmentation reduces the data processing burden by eliminating unnecessary derived fields from real-time processing, making it feasible to achieve high detection speeds while handling large data volumes. The system processes only the relevant segment of data needed for fraud detection.
Solution Approach 2:
The patent implements preliminary action by pre-establishing the derived field hierarchy and identifying root derived fields before real-time detection begins. This pre-computation creates a streamlined processing path that can be quickly traversed during fraud detection, reducing the difficulty of processing large data volumes in real-time. The preliminary structure enables rapid querying and processing without re-analyzing the entire data set.
Data Source
AI summary
A computer-implemented system, platform, programing product, and/or method for improving transformation selection in an ensemble machine learning (ML) model that includes: providing all base ML models of the ensemble ML model; identifying all of a plurality of Derived Fields in all the base ML models; performing a Derived Field run prediction analysis for all the Derived Fields; computing the Derived Field Importance Weight for Field (DFIW4F) and the Derived Field Importance Weight for Model (DFIW4M) for all the Derived Fields; clustering all the Derived Fields into a plurality of Derived Field clusters, wherein each Derived Field cluster is based upon the DFIW4M and the DFIW4F for the Derived Field; sorting all the Derived Field clusters by best cluster based upon DFIW4M and DFIW4F; and running the base ML models based upon the Derived Fields in the best Derived Field cluster until sufficient base ML models have been run.