Lineage Information Management in Data Analytics Workloads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing big data analytics and management systems lack the capability to effectively manage and store lineage information, which is crucial for data provenance, de-duplication, and workload scheduling, leading to inefficiencies such as outdated input data and redundant storage of similar analytics outputs.
Innovation Solution
A method is introduced to generate and manage lineage information within a data analytics and management system by creating a unique identifier for workloads, registering metadata associated with output datasets, and utilizing this metadata to track connections between data, enabling data provenance, de-duplication, and workload scheduling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing big data analytics systems are used, then scalability and reliability are improved, but the capability to manage and store lineage information is lost
Solution Approach 1:
The patent extracts lineage information management as a separate functionality from the core big data analytics system. A dedicated metadata store is introduced to capture and store lineage information independently, allowing the system to maintain scalability while preserving data provenance, de-duplication, and scheduling information that would otherwise be lost.
2Loss of information
If lineage information management is added to big data systems, then data provenance and de-duplication capabilities are improved, but system complexity increases
Solution Approach 1:
The patent introduces a metadata store as an intermediary component between the data analytics workloads and the lineage information requirements. This mediator captures lineage information automatically during workload execution without requiring complex changes to the core analytics engine, thereby reducing overall system complexity while enabling comprehensive lineage tracking.
3Loss of information
If metadata registration is implemented for all output datasets, then data connection tracing is improved, but processing time increases
Solution Approach 1:
The patent implements preliminary action by capturing and registering metadata information during the data analytics workload execution process itself, rather than performing separate post-processing to trace data connections. The lineage information is recorded as part of the normal workflow, eliminating the need for additional processing time after analytics completion while maintaining comprehensive data connection tracing capability.
Data Source
AI summary
A data analytics workload is obtained, wherein the data analytics workload includes one or more execution parameters and an input data set. An identifier specific to the data analytics workload is generated. The data analytics workload is at least partially executed based on the one or more execution parameters and the input data set to generate an output data set. Meta data associated with the output data set generated by execution of the data analytics workload is obtained, wherein the meta data includes lineage information corresponding to the output data set generated by execution of the input data set. The meta data is registered in a meta data store.