Dynamic Predictive Modeling In-Memory Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Financial institutions face significant challenges in managing and processing massive amounts of data for stress testing and predictive modeling, leading to bottlenecks in data loading and processing due to redundant data movement between storage and processing units.
Innovation Solution
Implementing a dynamic predictive modeling system that accesses raw data in memory, allowing the CPU to perform computations directly where the data resides, eliminating redundant information movement and reducing bottlenecks by using snapshot logic to prepare data for modeling and selecting subsets of forecast pairs based on predetermined increments of time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is moved between storage and processing units for stress testing and predictive modeling, then the modeling can be performed, but data loading and processing become bottlenecked due to redundant data movement
Solution Approach 1:
The patent combines storage and processing functions by implementing in-memory computing where data is loaded once into memory and processed multiple times without repeated I/O operations. This merging eliminates the bottleneck of redundant data movement between storage and processing units, allowing the same data to be accessed and processed repeatedly in memory.
Solution Approach 2:
The patent pre-loads data into memory before processing begins, and pre-computes certain data structures and intermediate results. This preliminary action ensures that when processing starts, the data is already in an optimal state for computation, eliminating delays during the actual modeling process.
2Reliability
If massive amounts of data are stored and manipulated for stress testing, then comprehensive modeling is achieved, but substantial and increasing amounts of storage and processing power are required
Solution Approach 1:
The patent segments the data processing into distinct phases: data loading phase, in-memory processing phase, and result output phase. It also segments the computational workload by processing different data subsets in parallel and using incremental updates. This segmentation reduces the peak memory and processing requirements while maintaining model accuracy.
Solution Approach 2:
The patent changes the state of data from persistent storage format to in-memory computational format. It transforms raw data into pre-processed structures suitable for statistical analysis, changing parameters such as data representation, precision, and organization to optimize for speed and memory efficiency rather than storage capacity.
3Adaptability or versatility
If data is loaded and processed repeatedly for different models and scenarios, then comprehensive stress testing is performed, but data traffic and processing time increase
Solution Approach 1:
The patent creates copies of data in optimized in-memory formats suitable for different types of analysis. Instead of repeatedly loading from storage, it maintains multiple in-memory representations of the data that can be accessed instantly for different modeling scenarios, reducing I/O traffic while maintaining versatility.
Solution Approach 2:
The patent implements a universal in-memory data structure that serves multiple processing functions simultaneously. The same loaded data can be used for various statistical analyses, model types, and stress testing scenarios without requiring separate loading operations, making the system both versatile and efficient.
Data Source
AI summary
Dynamic predictive modelling is disclosed. A subset of joined pairs can be selected from a table of joined pairs in which each pair includes snapshot time and forecast time, the joined pairs form a forecasting triangle that spans predetermined increments of time, and the subset of joined pairs comprises a vector in the forecasting triangle. A predictive model can be utilized to operate over the data associated with the subset of joined pairs and produce a prediction. The subset of joined pairs can be dependent on the predictive model. In one instance, data associated with the subset of pairs can be loaded into processor cache to facilitate efficient processing.