Real-Time Data Dependency Graph for Granular Task Orchestration

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional margin management workflow tools are inefficient in handling and processing vast amounts of data, often leading to delayed data sourcing and incorrect reporting due to their batch processing configuration, which fails to identify real-time data dependencies at a granular level.

Innovation Solution

A real-time data dependency management module that utilizes processors and memory to receive data sets from upstream applications, extract data entity events, identify dependent data entities through a data dependency graph, publish events for task orchestration, and execute tasks based on availability, enabling granular-level processing and accurate reporting.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If batch processing configuration is used, then processing capacity is improved, but data dependency identification accuracy deteriorates

Engineering Contradiction:

Improveprocessing capacityVSAvoiddata dependency identification accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The patent segments the batch processing workflow into individual portfolio-level processing units. Each portfolio is processed separately with its own data dependency graph, allowing granular identification of data dependencies while maintaining overall processing capacity. This segmentation enables the system to track data availability at the portfolio level rather than aggregating all portfolios into a single batch operation.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces a new dimension of data dependency tracking by implementing data dependency graphs at the portfolio level. This adds a granular dimension to the processing architecture, enabling simultaneous batch-level resource management and portfolio-level dependency identification. The system operates in two dimensions: batch processing for throughput and individual portfolio graphs for accuracy.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

2Use of energy by moving object

If batch processing is used, then resource utilization is improved, but reporting timeliness deteriorates

Engineering Contradiction:

Improveresource utilizationVSAvoidreporting timeliness

Core Design Contradiction:

Use of energy by moving objectVSLoss of time

Solution Approach 1:

The patent implements dynamic portfolio-level data dependency graphs that can be initialized and processed independently. This dynamic approach allows the system to process portfolios as soon as their specific data dependencies are met, rather than waiting for a batch window. The system dynamically adjusts processing based on data availability at the portfolio level while maintaining efficient resource utilization through controlled concurrency.

Inventive Principle:

Principle #15Dynamics

3Device complexity

If batch processing configuration is used, then system complexity is reduced, but data sourcing accuracy deteriorates

Engineering Contradiction:

Improvesystem complexityVSAvoiddata sourcing accuracy

Core Design Contradiction:

Device complexityVSManufacturing precision

Solution Approach 1:

The patent segments data sourcing into portfolio-specific operations, where each portfolio has its own data dependency graph tracking upstream data sources. This segmentation enables accurate tracking of data provenance and availability for each portfolio independently, ensuring data sourcing accuracy while keeping the overall system manageable through modular portfolio-level processing units.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS12118385B2System and method for real-time data dependency management and task orchestration

Publication Date: 2024.10.15 JPMORGAN CHASE BANK NA

AI summary

Various methods, apparatuses/systems, and media for real-time data dependency management are disclosed. A processor extracts data entity events from a plurality of data sets from upstream application; identifies dependent data entities for each data entity event based on initializing a data dependency graph with parent data nodes that represent all entities and their respective child data dependencies; publishes a data dependency event for each required parent data node in the data dependency graph; publishes a data dependency ready event for a certain parent node based on determining the certain parent node is configured for event publishing and that the certain parent node's child data dependencies are available; transmits the data dependency ready event to a task orchestration service platform; and orchestrates, upon receiving the data dependency ready event by the task orchestration service platform, a process instance and executes tasks for a corresponding data entity based on the process instance.