Recursive Data Refinement System for Traceable Lineage

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional data processing and refinement methods in data lakes are typically one-time processes, making it difficult to trace the lineage of refined data and utilize transitional data at each stage of the refinement process.

Innovation Solution

A method and system for facilitating recursive refinement of data in a data lake by progressively processing, republishing, and documenting data, which includes receiving raw data, generating conformed and featurized data sets, and publishing them for further processing while maintaining data lineage information.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of information

If conventional one-time data processing and refinement methods are used, then data processing can be completed, but data lineage cannot be easily traced and transitional data are not available for use at each stage

Engineering Contradiction:

Improvedata lineage traceabilityVSAvoiddata processing system complexity

Core Design Contradiction:

Loss of informationVSDevice complexity

Solution Approach 1:

The patent implements continuous data refinement where processed data is republished back to the data lake, enabling ongoing processing cycles. This creates a continuous action loop that maintains data lineage information throughout multiple refinement iterations, allowing traceability while avoiding complex one-time processing architectures.

Inventive Principle:

Principle #20Continuity of useful action

Solution Approach 2:

The system publishes refined data back to the data ingestion component, creating a feedback loop. This feedback mechanism ensures that lineage information is preserved and propagated through subsequent processing cycles, enabling traceability without increasing system complexity.

Inventive Principle:

Principle #23Feedback

2Productivity

If recursive data refinement with republishing is implemented, then data usability and traceability improve, but processing time and system complexity increase

Engineering Contradiction:

Improvedata usability and availabilityVSAvoidprocessing time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The system performs preliminary data conforming and feature generation in real-time during the refinement process. By preparing data incrementally and publishing intermediate results, the system improves data usability without requiring complete reprocessing, thus reducing overall processing time despite multiple refinement cycles.

Inventive Principle:

Principle #10Preliminary action

3Reliability

If conventional one-time data refinement is used, then processing is simple, but refined data cannot be easily reproduced or replicated

Engineering Contradiction:

Improvedata reproduction and replication capabilityVSAvoiddata processing ease

Core Design Contradiction:

ReliabilityVSEase of manufacture

Solution Approach 1:

The system creates copies of refined data by publishing them back to the data lake in standardized formats. This copying mechanism enables easy reproduction and replication of refined data across different processing cycles and applications, while maintaining processing simplicity through automated republishing workflows.

Inventive Principle:

Principle #26Copying

Data Source

PatentUS12235812B2Method and system for recursive data refinement and republishing

Publication Date: 2025.02.25 JPMORGAN CHASE BANK NA

AI summary

A method for facilitating recursive data refinement with self-documentation is provided. The method includes receiving, via a data ingestion component, raw data that is published from a networked repository; generating a conformed data set from the raw data based on a conformation parameter; retrieving, via a communication interface, a feature configuration; generating, in real-time, a featurized data set from the conformed data set based on the feature configuration; and publishing the featurized data set to the data ingestion component for further processing.