Clustering Relevance Computation in Volatile Data Environments
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In volatile data environments, such as the bond market, predicting future data is challenging due to unstable relationships between data groups, limited trading throughput, and insufficient data traces for certain bonds, making conventional data models unsuitable for accurate predictions.
Innovation Solution
A method involving plotting a time series dataset, generating grand truth data values using a machine learning algorithm, clustering the dataset into multiple data clusters based on data correlation, training sub-machine learning algorithms for each cluster, predicting future data values, estimating clustering error, and adjusting the composition of data clusters to improve accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional data models are used for prediction in volatile environments, then the model structure remains simple, but prediction accuracy deteriorates due to unstable data relationships
Solution Approach 1:
The patent segments the data into multiple clusters based on correlation characteristics, and trains separate machine learning models for each cluster. This segmentation allows the system to handle volatile data relationships by treating different data groups differently, improving prediction accuracy without requiring a single overly complex model to handle all data uniformly.
Solution Approach 2:
The patent dynamically adjusts cluster compositions and model parameters based on changing data relationships over time. The system continuously monitors data correlation patterns and reassigns data points to clusters as relationships evolve, enabling the model to adapt to volatility while maintaining predictive accuracy through dynamic rather than static structure.
2Stability of the object's composition
If data is clustered to handle volatility, then prediction stability improves, but computational complexity increases due to multiple clustering and training operations
Solution Approach 1:
The patent performs preliminary clustering of data into stable groups before training machine learning models. By pre-organizing data into clusters based on correlation patterns, the system reduces the complexity of subsequent training operations and enables more efficient predictive modeling, as the clustering structure guides the learning process rather than requiring the model to discover patterns from raw data.
Solution Approach 2:
The patent changes parameters such as cluster assignment and model training parameters based on data characteristics and performance metrics. By adjusting these parameters dynamically, the system optimizes the balance between cluster stability and computational efficiency, allowing the use of multiple clustering operations without excessive computational burden.
3Reliability
If multiple sub-ML algorithms are trained for each cluster, then prediction accuracy for volatile data improves, but training time and computational resources increase
Solution Approach 1:
The patent applies different machine learning algorithms or parameters to different clusters based on their specific characteristics and volatility patterns. Rather than using a single uniform model for all data, the system tailors the modeling approach to each cluster's local properties, improving predictive accuracy for volatile data while managing training time by focusing computational resources where they are most needed.
Data Source
AI summary
A method and system for determining clustering relevance in a volatile data environment and adjusting clustering composition for improved accuracy are disclosed. The method includes plotting a dataset and generating at least one grand truth data value, and clustering the plotted dataset for generating data clusters, in which the clustering is performed based on correlation of individual data values included in the dataset. The method further includes independently training machine learning (ML) algorithm for each of the data clusters for generating a managing ML algorithm for the dataset, applying the managing ML algorithm to the dataset for predicting at least one future data value, and comparing differences between the grand truth data value and the future data value for estimating a clustering error, and adjusting composition of at least one of the plurality of data clusters based on the estimated clustering error.