Distributed Data Processing System with Auto-Recovery

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing financial instrument processing systems face inefficiencies in handling large volumes of documents, particularly in identifying and processing exceptions, due to the lack of effective distributed data processing and synchronization mechanisms, which can lead to delays and errors in reconciliation and recovery processes.

Innovation Solution

A data processing system that divides data processing jobs into smaller items and allocates them across a distributed network of user systems, using in-flight tables for monitoring and active synchronization to ensure efficient processing and automatic recovery in case of faults, by periodically saving progress and results, and reallocating work based on system capacity.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If data processing jobs are processed in a centralized system, then system simplicity is maintained, but processing speed and efficiency deteriorate when handling large volumes of documents

Engineering Contradiction:

Improveprocessing speedVSAvoidsystem complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent segments data processing jobs into smaller data processing items and distributes them across multiple user systems in a distributed network. The master queue divides jobs into manageable units that can be processed independently by different systems simultaneously, enabling parallel processing and improving overall processing speed without requiring a monolithic complex system

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent transitions from a single-dimensional centralized processing model to a multi-dimensional distributed processing architecture. By introducing spatial distribution across multiple user systems and organizing work through hierarchical queues (master queue, in-flight tables, work queues), the system achieves higher productivity by utilizing additional dimensional space for parallel operations

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

2Reliability

If distributed data processing is implemented without synchronization mechanisms, then processing efficiency improves through parallel operations, but data consistency and reliability deteriorate

Engineering Contradiction:

Improvedata consistencyVSAvoidsynchronization complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent implements feedback mechanisms through in-flight tables that continuously monitor the status of distributed data processing items. Each user system reports back its processing status, and the master queue receives these updates to maintain an accurate view of work in progress. This feedback loop ensures data consistency across the distributed network while managing synchronization through structured status tracking rather than complex inter-system coordination

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The patent introduces in-flight tables as intermediary structures between the master queue and distributed user systems. These tables serve as buffer and coordination points that manage the flow of data processing items, track their status, and facilitate synchronization without requiring direct complex communication between all distributed systems. The intermediary absorbs synchronization complexity while maintaining simplicity at the user system level

Inventive Principle:

Principle #24Intermediary (Mediator)

3Reliability

If data processing continues without interruption, then productivity is maximized, but system reliability deteriorates due to potential faults and errors

Engineering Contradiction:

Improvefault toleranceVSAvoidprocessing continuity

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent implements preliminary action through periodic saving of processing status and results to the in-flight tables before faults can occur. By continuously updating and persisting the state of data processing items during normal operation, the system prepares recovery information in advance, enabling quick restoration after faults without significant productivity loss. This proactive status preservation ensures fault tolerance while maintaining processing continuity

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent implements automatic recovery mechanisms that discard faulty processing states and recover from saved status information in the in-flight tables. When a fault is detected, the system can reprocess affected data processing items using the previously saved status and results, automatically recovering from errors without manual intervention. This approach maintains reliability by enabling fault recovery while preserving productivity through automated continuation of processing

Inventive Principle:

Principle #34Discarding and recovering

Data Source

PatentUS9952942B2System for distributed data processing with auto-recovery

Publication Date: 2018.04.24 BANK OF AMERICA CORP

AI summary

Embodiments enable distributed data processing with automatic caching at multiple system levels by accessing a master queue of data processing work comprising a plurality of data processing jobs stored in a long term memory cache; selecting at least one of the plurality of data processing jobs from the master queue of data processing work; pushing the selected data processing jobs to an interface layer including (i) accessing the selected data processing jobs from the long term memory cache; and (ii) saving the selected data processing jobs in an interface layer cache of data processing work; and pushing at least a portion of the selected data processing jobs to a memory cache of a first user system for minimizing latency in user data processing of the pushed data processing jobs.