Data Lineage Trees Using Policy Exceptions to Cut Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data management systems face challenges in efficiently storing and retrieving data lineage information due to the rapid growth of lineage records with each copy, leading to scalability issues and increased storage requirements, especially in complex storage environments.
Innovation Solution
A method that utilizes policy and exception information to dynamically generate data lineage views by constructing lineage trees based on backup policies, rather than storing individual lineage records, and incorporates a Recovery-Failure-Delete model to track exceptions and changes, enabling efficient real-time lineage tracking.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If individual lineage records are stored for each copy of data assets, then complete lineage tracking is achieved, but storage requirements and system complexity grow rapidly
Solution Approach 1:
The patent merges multiple individual lineage records into a single consolidated lineage record that represents an entire lineage tree. Instead of storing separate records for each copy of a data asset, the system creates one record that captures the hierarchical relationships of all copies, thereby reducing storage requirements while maintaining complete lineage tracking capability
Solution Approach 2:
The patent uses symbolic copying where a single lineage record represents multiple actual data copies through hierarchical relationships. The consolidated record contains references to parent and child nodes that symbolically represent the entire lineage structure, allowing the system to track numerous copies without proportionally increasing storage requirements
2Reliability
If individual lineage records are stored for each copy of data assets, then complete lineage tracking is achieved, but system complexity increases
Solution Approach 1:
The patent consolidates the management of multiple lineage records into a single hierarchical structure. By merging individual records into one consolidated lineage record with parent-child relationships, the system reduces the number of discrete objects that need to be managed, indexed, and queried, thereby lowering system complexity
Solution Approach 2:
The patent segments the lineage information into hierarchical levels (parent nodes and child nodes) within a single consolidated record. This segmentation allows the system to manage complexity by organizing information in a structured tree format rather than as a flat collection of individual records, making it easier to traverse and query lineage relationships
3Speed
If lineage information is stored in real-time for numerous copies, then rapid lineage search is enabled, but storage requirements increase linearly with number of copies
Solution Approach 1:
The patent merges all lineage information for numerous copies into a single consolidated record, eliminating the need to store and retrieve individual records for each copy. This merging enables rapid lineage search by providing a unified view of the entire lineage tree without requiring linear scaling of storage with the number of copies
Solution Approach 2:
The patent transitions from a flat, one-dimensional storage model where each copy requires a separate record to a hierarchical, multi-dimensional structure where a single record contains nested parent-child relationships. This dimensional change allows the system to represent exponential lineage relationships using linear storage space
Data Source
AI summary
A computer-implemented system and method can be used to identify policy information applicable to a compute resource within a specified time range, the policy information related to data management for the compute resource. Based on the identified policy information, a set of expected data management events for the compute resource is generated within the specified time range. The set of expected data protection events is modified based on information related to exceptions to the data management indicated by the policy information. The modified set of data protection events provides lineage information for the compute resource.