Data Object Graph Enhancement for Automated Lineage Discovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The complexity and rapid growth of data volumes across various platforms and systems, coupled with changing business requirements, make it difficult to understand and leverage the relationships between data objects, leading to inefficiencies and missed opportunities for process improvements.
Innovation Solution
A framework that identifies relationships between data objects by comparing lineage data objects with candidate data objects using metadata and attributes, establishing relationships based on satisfied criteria, and rendering an updated lineage graph for display.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data relationships are manually tracked and documented, then relationship accuracy can be maintained, but the complexity and time required to manage thousands of data objects increases significantly
Solution Approach 1:
The system enables data objects to automatically generate and maintain their own relationship metadata through self-description mechanisms. Each data object contains inherent metadata that describes its relationships with other data objects, eliminating the need for manual relationship tracking and documentation while maintaining accuracy.
Solution Approach 2:
The patent replaces manual mechanical processes of relationship documentation with automated computational systems. The framework uses algorithms to automatically discover, validate, and update data relationships by analyzing metadata and data patterns, substituting human effort with automated relationship inference engines.
2Loss of information
If comprehensive data relationship tracking is implemented across all data objects, then complete understanding of data interconnections is achieved, but the complexity of the system increases
Solution Approach 1:
The system segments the complex task of relationship tracking by implementing a hierarchical metadata structure. Relationships are organized into different levels of abstraction, with core relationships at higher levels and detailed relationships at lower levels, allowing comprehensive tracking without overwhelming system complexity.
Solution Approach 2:
The patent introduces metadata as an intermediary layer between raw data objects and relationship analysis. This metadata layer acts as a mediator that captures relationship information without requiring direct complex interactions between all data objects, simplifying the overall system architecture while maintaining comprehensive relationship understanding.
3Reliability
If traditional ETL processes are used to share data between systems, then data accuracy and consistency can be ensured, but the processes become complex to design and implement
Solution Approach 1:
The system enables data objects to automatically generate and maintain their own relationship metadata through self-description mechanisms. Each data object contains inherent metadata that describes its relationships with other data objects, eliminating the need for manual relationship tracking and documentation while maintaining accuracy.
Solution Approach 2:
The patent transforms the approach to data sharing by changing the parameters of data object descriptions. Instead of using traditional ETL transformation parameters, the system uses enriched metadata parameters that inherently describe relationships, making the data sharing process simpler while maintaining accuracy and consistency.
Data Source
AI summary
Techniques and solutions are provided for identifying relationships between data objects. In particular, the present disclosure provides techniques that can identify data objects that are related to data objects in a lineage graph, as well as being useable to identify relationships between existing lineage graph data objects. A lineage data object is compared with a candidate data object. For example, attribute types, attribute values, or information regarding such attribute types or values can be compared between the lineage data object and the candidate data object. If it is determined that a relationship criterion is satisfied, a relationship can be established or suggested between the lineage data object and the candidate data object. A display is rendered that displays a lineage graph with the relationship between the lineage data object and the candidate data object. A user can define an edge between the lineage data object and the candidate data object.