Cross-Platform Data Lineage Graphs for Streaming Transformation Tracing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional data lineage and classification processes lack scalability and support for complex data events across multiple systems, leading to vendor lock-ins and limitations in capturing provenance and lineage traceability, especially in data streaming and offline transformations.

Innovation Solution

A method and system for tracing data across platforms, involving devices that provide context data, store it in a database, extract and process lineage tracer blocks, convert them into linked triples, and generate a lineage graph for visualization, supporting both streaming and offline transformations.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If conventional data lineage and classification processes are used, then data accuracy and reliability are ensured, but scalability is limited and vendor lock-ins occur

Engineering Contradiction:

Improvedata accuracyVSAvoidscalability

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

The system segments data lineage tracking into modular components: context data collection from multiple devices, database storage, data classifier block extraction, lineage tracer block processing, and visualization generation. This modular architecture enables independent scaling of each component while maintaining data accuracy through structured processing pipelines.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The lineage processing system is designed to handle multiple data types and transformation operations universally. The data classifier and lineage tracer can process streaming data, batch data, and offline transformations through a unified framework that supports various data formats and processing modes without vendor-specific dependencies.

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Measurement precision

If conventional solutions are used, then data classification is performed, but support for complex data events across multiple systems is lacking

Engineering Contradiction:

Improvedata classificationVSAvoidsupport for complex data events

Core Design Contradiction:

Measurement precisionVSAdaptability or versatility

Solution Approach 1:

The system introduces intermediary components including a database that stores context data from multiple devices, a data classifier that extracts classification blocks, and a lineage processor that generates lineage triples. These intermediaries enable precise data classification while bridging complex data events across different systems and platforms through standardized processing interfaces.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system adds dimensional depth to data classification by creating multi-layered lineage information. Context data from multiple devices is processed through classification and tracing to generate lineage graphs that represent data flow across spatial (multiple systems) and temporal (streaming and batch processing) dimensions, enabling comprehensive tracking of complex data events.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

3Loss of information

If conventional lineage processing is used, then data provenance is captured, but offline transformations are not supported

Engineering Contradiction:

Improveprovenance captureVSAvoidoffline transformation support

Core Design Contradiction:

Loss of informationVSAdaptability or versatility

Solution Approach 1:

The lineage processing system dynamically adapts to different transformation types. The lineage processor can handle both real-time streaming transformations and offline batch transformations by adjusting its processing mode. Context data is collected and stored, then processed according to the specific transformation type, enabling flexible support for diverse data processing scenarios while maintaining provenance information.

Inventive Principle:

Principle #15Dynamics

4Reliability

If conventional solutions are used, then data tracing is performed, but vendor dependencies limit system flexibility

Engineering Contradiction:

Improvedata tracingVSAvoidsystem flexibility

Core Design Contradiction:

ReliabilityVSEase of operation

Solution Approach 1:

The system changes key parameters by implementing an open, standardized data model for lineage tracking. Context data is stored in a vendor-neutral database format, classification and tracing operations use standardized algorithms, and lineage graphs are generated in universal visualization formats. This parameter change from proprietary to standardized approaches maintains reliable data tracing while eliminating vendor dependencies and improving system flexibility.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS12443507B2System and method for tracing data streamed across different platforms and identifying data manipulations performed across different platforms

Publication Date: 2025.10.14 JPMORGAN CHASE BANK NA

AI summary

A method and system for tracing data streamed across differing different system platforms are disclosed. The method includes providing and storing context data corresponding to a data event published to a streaming service, extracting a data classifier block from the stored context data, and extracting a lineage tracer block from the stored context data. The method further includes converting the lineage tracer block into a linked lineage triple, and generating a lineage graph using the linked lineage triple for visualization.