Data Lineage Tracking for Big Data Transformation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional relational databases struggle to handle large volumes of big data, leading to difficulties in tracking data lineage, detecting duplicate data transformations, and managing data storage across various formats, especially when dealing with sensitive information.
Innovation Solution
A system and method for automatically recording and storing lineage data in a centralized repository, which includes transformation logic and source variables, allowing for the detection of duplicate data transformations and efficient data management across multiple big data storage formats.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is stored in numerous different data storage formats in various locations, then diverse application parameters and use case parameters can be serviced, but it becomes difficult and time consuming to retrace the layers of transformations for a given variable
Solution Approach 1:
The system performs preliminary action by automatically capturing and recording lineage data at the time data transformations occur. Metadata is generated and stored in a centralized repository during the ETL process, so when lineage information is needed later, it is already available without requiring time-consuming retroactive analysis of transformation layers.
2Reliability
If traditional relational databases are used, then data can be stored in structured formats, but they are not capable of sufficiently handling the size of the tables that big data creates
Solution Approach 1:
The system segments the data storage architecture by separating the big data storage layer (Hadoop distributed file system) from the lineage metadata storage layer (relational database). This segmentation allows each component to be optimized independently - the distributed file system handles large-scale data storage while the relational database manages structured lineage information, avoiding the limitations of using a single database system for both purposes.
3Adaptability or versatility
If data transformations are performed through several layers, then diverse output formats can be generated, but duplicative output data is difficult to detect and prevent
Solution Approach 1:
The system implements feedback by continuously capturing lineage information and using it to detect duplicative transformations. The centralized repository stores metadata that provides feedback about existing data transformations, allowing the system to identify when duplicate output data is being generated and prevent redundant processing.
4Loss of information
If lineage data is manually tracked, then data transformations can be documented, but the process becomes time consuming and error prone
Solution Approach 1:
The system enables self-service by automatically generating and maintaining lineage metadata without requiring manual intervention. The ETL framework automatically captures transformation information, source data identifiers, and destination data locations, eliminating the need for manual lineage tracking while ensuring complete and accurate documentation of all data transformations.
Data Source
AI summary
Systems and methods of the present disclosure may read source data corresponding to a source variable; apply a first transformation to the source variable to generate a first output variable; record lineage data of the first output variable, wherein the lineage data includes the first transformation, the source variable, and the first output variable; receive a request to generate a second output variable from the source data corresponding to the source variable; determine a second transformation to be applied to the source variable to generate the second output variable; match the second transformation with first transformation using the lineage data; and/or output the first output variable in place of generating the second output variable.