Hierarchical Predictive Model Building for Large Datasets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Building optimal predictive models for large datasets is challenging due to computational demands and the presence of many local optimal solutions, which can lead to discrepancies in model performance, especially in high-volume and high-dimensional data environments.
Innovation Solution
A hierarchical method is introduced that preprocesses datasets, clusters them hierarchically, and builds predictive models in a layered approach, allowing for the computation of multiple local optimal solutions and improving model performance by leveraging group properties within the data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional model building techniques are applied to large datasets, then model performance can be achieved, but computational complexity and time requirements increase significantly
Solution Approach 1:
The patent divides the dataset into multiple clusters using hierarchical clustering, and builds multiple predictive models for different clusters rather than building a single model for the entire dataset. This segmentation reduces the computational complexity of each individual model while maintaining overall model performance through ensemble methods.
Solution Approach 2:
The patent introduces a hierarchical clustering dimension to the traditional model building process. By organizing data into multiple levels of clusters, the method adds a structural dimension that enables more efficient computation while preserving the relationships between data points, thus reducing computational complexity without sacrificing performance.
2Reliability
If traditional optimization methods are used to find the best model parameters, then model performance can be optimized, but the methods may get trapped in local optimal solutions
Solution Approach 1:
The patent applies segmentation by building multiple predictive models for different clusters identified through hierarchical clustering. Each cluster can be optimized independently, and the ensemble of models provides a more comprehensive search of the parameter space, reducing the likelihood of getting trapped in local optima and improving the ability to find globally optimal solutions.
Solution Approach 2:
The patent changes the approach to parameter optimization by using multiple models with different parameters trained on different cluster subsets. This diversification of parameters across multiple models enables a more thorough exploration of the optimization landscape, helping to avoid local optima and improve adaptability in finding the best overall model performance.
3Reliability
If multiple predictive models are built for different clusters, then model performance and generalization improve, but the overall computational time increases
Solution Approach 1:
The patent performs preliminary hierarchical clustering to organize the data into clusters before building the predictive models. This preliminary organization groups similar data points together, enabling more efficient model training by reducing the search space and allowing parallel processing of multiple cluster models, thus improving generalization while controlling computational time.
Solution Approach 2:
By segmenting the dataset into clusters and building models for each cluster independently, the patent enables parallel computation. This segmentation allows multiple models to be trained simultaneously, which reduces the total computational time required compared to building a single monolithic model, while still achieving improved generalization through the ensemble of specialized models.
Data Source
AI summary
Predictive analytic models are hierarchically built based on a training dataset, which includes pairs of input data and output data. First, the input data and the output data are preprocessed. A hierarchical clustering process is performed on the dataset. The hierarchical clustering process comprises level-1 input and output data clustering, level-2 input and output data clustering, and so on, up to level-K input and output data clustering, where K is an integer greater than one. A hierarchical model building process is performed. The hierarchical model building process comprises level-1 model building over level-1 clustered input and output data, level-2 model building over level-2 clustered input and output data, and so on, up to level-K model building over level-K clustered input and output data. At least one level-K predictive model is generated as the resulting built model.