Inter-Dependent Data Arrays for Faster Risk Model Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing statistical analysis systems face inefficiencies in processing inter-dependent data, particularly in large-scale credit risk modeling and analysis, where multithreading does not improve performance due to rows not being processed together, leading to prolonged run times.
Innovation Solution
A system and method that processes inter-dependent data using a logical array, converting columns with dependencies into a single column for parallel processing, utilizing user-defined functions and an explode function to handle dependencies efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If row-by-row data processing is used in statistical analysis systems, then data processing can be performed sequentially, but processing time is prolonged and multithreading does not improve performance due to row dependencies
Solution Approach 1:
The patent transforms the data structure from a two-dimensional table (rows and columns) to a one-dimensional array. This dimensional change allows the system to process inter-dependent data in a linear sequence that preserves row dependencies while enabling efficient single-threaded processing, eliminating the need for complex multi-threading coordination and significantly reducing processing time.
Solution Approach 2:
Instead of processing data row-by-row as traditionally done in statistical analysis systems, the patent inverts the approach by converting the entire table to an array and processing it in a linear fashion. This inversion of the processing paradigm allows the system to handle row dependencies efficiently without the performance penalties associated with traditional row-by-row or multi-threaded approaches.
2Productivity
If data is organized into partitions of a relational database for distributed processing, then data can be spread over clusters, but processing inter-dependent data across clusters remains inefficient
Solution Approach 1:
The patent applies dimensional transformation by converting the multi-dimensional relational database structure into a one-dimensional array representation. This simplifies the data organization for distributed processing, as the linear array structure can be more efficiently partitioned and distributed across clusters while maintaining the inter-dependency relationships through the array's sequential nature.
Data Source
AI summary
A system for processing data items may include one or more memories and one or more processors. The one or more memories may be configured in a first logical table including a plurality of logical rows and a plurality of logical columns. A value of a data item in a first logical column in each logical row may be determined based on a dependency of the first logical column on a second logical column in another logical row. The one or more processors may configure at least the first and second logical columns of the first logical table into a first logical array of data items, determine, by executing a first execution unit, values of the data items in the first logical array using the dependency, and convert, by executing the first execution unit, the first logical array with the determined values into a second logical table.