Distributed Ledger Data Lineage Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems require significant manual effort to trace data in motion between applications, which leads to low data quality due to incomplete knowledge of data flows among application team members, and this effort needs to be repeated regularly to maintain data provenance.
Innovation Solution
The use of distributed ledgers, specifically a system and method for tracking data lineage and record lifecycle, where a record lifecycle tool creates a recordable artifact for a record lifecycle event, generates a hash, creates metadata, signs it, and writes it to an immutable supply chain metadata storage, with the hash and storage location identifier written to a present state database.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If manual tracking of data flows is performed to meet regulatory requirements, then data lineage information can be obtained, but the data quality is low and the process requires significant manual effort
Solution Approach 1:
The system automatically captures data lineage information at the source systems when data is created or moved, rather than requiring manual tracking later. Event listeners and agents are deployed to proactively record data flows as they occur, eliminating the need for manual intervention to trace data provenance.
Solution Approach 2:
A centralized data lineage platform acts as an intermediary between source systems and users. This platform receives automated data flow events from multiple sources, processes them, and provides unified lineage information, reducing manual effort by serving as a single point of query and management.
2Loss of information
If manual declaration of data flows by responsible individuals is used, then data flow information can be collected, but data quality is low due to incomplete knowledge
Solution Approach 1:
Source systems automatically self-report their data flows to the lineage platform through deployed agents and event listeners. The systems themselves generate and submit data flow events without human intervention, ensuring complete and accurate information is captured directly at the source.
Solution Approach 2:
The system implements continuous monitoring where data flow events are captured, processed, and stored in real-time. This creates a feedback loop that automatically updates lineage information as data moves through systems, ensuring the information remains current and complete without manual updates.
3Reliability
If manual tracking is repeated regularly to keep data provenance current, then data lineage remains evergreen, but significant time and effort are consumed
Solution Approach 1:
The system continuously monitors and captures data flows in real-time through deployed agents and event listeners that operate continuously across source systems. This continuous automatic tracking eliminates the need for periodic manual updates, keeping data provenance current without interrupting business operations.
Data Source
AI summary
Systems and methods for tracking data lineage and record lifecycle using a distributed ledger are disclosed. In one embodiment, a method for tracking record lifecycle events may include: (1) creating, by a record lifecycle tool, a recordable artifact for a record lifecycle event in a record lifecycle, the recordable artifact comprising data for the record lifecycle event; (2) generating, by the record lifecycle tool, a hash of the data; (3) creating, by the record lifecycle tool, record lifecycle event metadata for the recordable artifact; (4) signing, by the record lifecycle tool, the record lifecycle event metadata; (5) writing the record lifecycle event metadata to supply chain metadata storage at a storage location, wherein the supply chain metadata storage may be cryptographically verifiable and immutable; and (6) writing the hash and an identifier for the storage location in the supply chain metadata store to a present state database.