Inter-Dependent Data Arrays for Distributed Risk Modeling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing statistical analysis systems struggle to efficiently process inter-dependent data, such as loan amortization schedules, due to row-by-row processing that does not benefit from multithreading, leading to prolonged processing times for large-scale credit risk modeling and analysis.
Innovation Solution
Process inter-dependent data using a logical array, converting columns with dependencies into a single column for column-wise calculations, and utilizing user-defined functions (UDFs) and an explode function to handle dependencies efficiently, enabling distributed computations over clusters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If row-by-row data processing is used in statistical analysis systems, then the system can process data in a simple sequential manner, but the processing time is excessively long for inter-dependent data because multithreading cannot be effectively utilized
Solution Approach 1:
The patent segments the inter-dependent data processing into independent computational units that can be distributed across multiple threads. By transforming the data representation and using array-based operations, the system divides the processing task into parallelizable segments while maintaining the required row dependencies through careful data organization and computation scheduling.
2Productivity
If data is organized in partitions of a relational database for distributed processing, then data can be spread over clusters to improve performance, but inter-row dependencies make efficient distribution difficult
Solution Approach 1:
The patent applies dimensionality change by transforming the traditional row-based relational database structure into an array-based representation. This dimensional transformation allows data to be organized in a way that preserves inter-row dependencies while enabling efficient distribution across clusters. The array structure provides a new dimension for data organization that facilitates both dependency management and parallel processing.
Data Source
AI summary
A system for processing data items may include one or more memories and one or more processors. The one or more memories may be configured in a first logical table including a plurality of logical rows and a plurality of logical columns. A value of a data item in a first logical column in each logical row may be determined based on a dependency of the first logical column on a second logical column in another logical row. The one or more processors may configure at least the first and second logical columns of the first logical table into a first logical array of data items, determine, by executing a first execution unit, values of the data items in the first logical array using the dependency, and convert, by executing the first execution unit, the first logical array with the determined values into a second logical table.