Distributed Data Processing with Multi-Level Automatic Caching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Financial instrument processing systems face inefficiencies in reconciling and processing large volumes of documents, particularly in identifying exceptions and minimizing latency, due to the complexity of handling multiple nodes or modules and the need for effective caching strategies.
Innovation Solution
A data processing system with automatic caching at multiple system levels, utilizing a master queue, interface layer caching, and in-flight data tables to prioritize and allocate data processing jobs across user systems, minimizing latency and optimizing resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data processing jobs are distributed across multiple nodes or modules, then processing capacity and reliability are improved, but system complexity and latency increase
Solution Approach 1:
The patent segments the data processing system into distinct functional modules including a master queue manager, interface layer with cache, in-flight data tables, and multiple user systems. Each module handles specific tasks independently, allowing parallel processing while maintaining clear boundaries that reduce overall system complexity despite distributed architecture
Solution Approach 2:
The interface layer cache acts as an intermediary between the master queue and user systems, buffering data processing jobs and reducing direct communication overhead between components. This mediator layer simplifies the interaction complexity while enabling efficient distributed processing across multiple nodes
2Speed
If data is cached at multiple system levels, then access speed and latency are improved, but memory usage and system complexity increase
Solution Approach 1:
The patent implements caching with different characteristics at different system levels: the interface layer cache uses one caching strategy while in-flight data tables use another. Each cache level is optimized for its specific function, allowing efficient memory utilization tailored to local access patterns rather than uniform caching throughout the system
Solution Approach 2:
The caching architecture nests multiple cache levels within each other, with the interface layer cache containing references to data that may also exist in in-flight data tables and master queue. This nested structure allows data to be accessed at multiple levels simultaneously, improving speed while managing memory usage through hierarchical organization
3Productivity
If priority-based job selection is implemented, then processing efficiency is improved, but computational overhead increases
Solution Approach 1:
Data processing jobs are assigned priority levels and organized in the master queue before being processed. This preliminary classification of jobs by priority allows the system to efficiently select which jobs to process next without performing complex computations during the actual processing phase, reducing real-time computational overhead
Data Source
AI summary
Embodiments enable distributed data processing with automatic caching at multiple system levels by accessing a master queue of data processing work comprising a plurality of data processing jobs stored in a long term memory cache; selecting at least one of the plurality of data processing jobs from the master queue of data processing work; pushing the selected data processing jobs to an interface layer including (i) accessing the selected data processing jobs from the long term memory cache; and (ii) saving the selected data processing jobs in an interface layer cache of data processing work; and pushing at least a portion of the selected data processing jobs to a memory cache of a first user system for minimizing latency in user data processing of the pushed data processing jobs.