Parallel Data Ingestion Engine for Regulatory Compliance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Legacy systems used by financial institutions for regulatory reporting are inefficient in handling global data, requiring extensive processing time and resources, which makes it challenging to meet strict regulatory timelines and increases overhead costs.
Innovation Solution
A dynamic ingestion method and system utilizing a parallel computation engine with multiple partitioned processors and a business logic processor pool to process large volumes of data in parallel, allowing for dynamic adjustment of processing power and significantly reducing processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If legacy batch processing systems are used to process global holdings data, then regulatory reporting requirements are met, but processing time increases to 12-14 hours and overhead costs increase
Solution Approach 1:
The system segments the global holdings data into regional partitions (North America, EMEA, APAC) and processes each partition independently using dedicated processors. This segmentation enables parallel processing of multiple regions simultaneously, reducing total processing time from 12-14 hours to approximately 1 hour while maintaining regulatory compliance for each region
Solution Approach 2:
The system dynamically adjusts processing resources by activating only the regional processors needed for current regulatory requirements. The architecture allows dynamic allocation of computing power to specific regions based on data volume and regulatory priorities, optimizing resource utilization and reducing overhead costs
2Adaptability or versatility
If legacy systems process data for multiple regions sequentially, then comprehensive regulatory coverage is achieved, but processing time extends to 12-14 hours
Solution Approach 1:
The system divides global holdings data into distinct regional partitions with dedicated processors for each region (North America, EMEA, APAC). This segmentation enables simultaneous parallel processing of multiple regions, increasing overall productivity while maintaining comprehensive regulatory coverage across all jurisdictions
Solution Approach 2:
The parallel processing architecture provides a universal framework that can handle regulatory reporting requirements for multiple regions simultaneously. The system is designed to be adaptable to different regulatory protocols and data formats across various jurisdictions, maintaining versatility while dramatically improving processing throughput
3Reliability
If legacy application framework is used, then regulatory reporting is completed, but infrastructure overhead and costs increase
Solution Approach 1:
The system implements dynamic resource allocation where computing resources are activated only when and where needed for specific regional processing tasks. This dynamic approach reduces infrastructure overhead by eliminating the need for continuous full-system operation, lowering energy consumption and operational costs while ensuring reliable reporting completion
Solution Approach 2:
Each regional processor independently manages its own data processing and regulatory compliance requirements without requiring centralized coordination for every operation. This self-service capability reduces the overhead burden on central infrastructure while maintaining reliable reporting completion across all regions
Data Source
AI summary
A method is provided for dynamic ingestion of data. The method includes detecting presence of data sets in an input queue, each data set belonging to a group. The method additionally includes fetching the data sets from each group utilizing a parallel computation engine including multiple partitioned processors, wherein each group is assigned to a partition of one of the multiple partitioned processors for processing. The method further includes processing the data sets for each group utilizing a group level process of the multiple partitioned processors to obtain resultant data sets and transmitting the resultant data sets for each group to a business logic processor pool in order to process the data sets in accordance with existing rules in order to generate a final data set.