Dataset Identification Using Canonical and Alias Graph Linking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data processing systems face inefficiencies in representing complex and interconnected data relationships due to duplication of datasets with multiple possible representations, leading to excessive storage utilization and inaccurate information retrieval.
Innovation Solution
A data lineage system processes unique identifiers collectively and individually to generate composite and alias identifiers, reducing duplicate graph nodes by linking new datasets to existing nodes in a graph representation, thereby maintaining accurate data relationships.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a data processing system uses a graph representation with nodes and edges to store complex and interconnected data relationships, then efficient storage and recall of information regarding complex relationships is achieved, but duplication of datasets with multiple possible representations leads to excessive storage utilization
Solution Approach 1:
The patent merges multiple representations of the same dataset by introducing a canonical identifier that uniquely identifies the dataset across different representations. When a dataset is encountered with multiple possible representations (e.g., different file paths, formats, or access methods), the system generates a single canonical identifier and stores all representations as references to this single identifier, thereby eliminating duplication while preserving all access pathways.
Solution Approach 2:
Instead of storing multiple complete copies of datasets with different representations, the patent creates lightweight reference copies (canonical identifiers) that point to the single master copy. This allows the system to maintain multiple access points and representations without duplicating the actual data content, significantly reducing storage utilization while maintaining retrieval accuracy.
2Adaptability or versatility
If a data processing system stores multiple representations of the same dataset in a graph structure, then all possible access methods are preserved, but redundant graph nodes are created leading to inaccurate information retrieval
Solution Approach 1:
The patent segments the dataset identification into two distinct components: the canonical identifier (which uniquely identifies the dataset) and the representation metadata (which describes how to access the dataset). This segmentation allows the system to maintain multiple access methods through different representation metadata while ensuring all point to the same canonical identifier, preventing redundancy and ensuring accuracy.
Solution Approach 2:
The canonical identifier acts as an intermediary between multiple dataset representations and the actual data storage. Instead of creating separate graph nodes for each representation, the system uses the canonical identifier as a mediator that links all representations to the single underlying dataset, eliminating redundant nodes while preserving all access pathways.
3Ease of operation
If the system generates separate graph nodes for each dataset representation, then all data access paths are maintained, but excessive duplicate nodes increase storage requirements and processing overhead
Solution Approach 1:
The canonical identifier serves multiple functions simultaneously: it uniquely identifies the dataset, acts as a key in the hash table for efficient retrieval, and serves as the graph node identifier. This multi-functionality eliminates the need for separate structures to handle identification and graph node management, reducing overall system complexity while maintaining access flexibility.
Solution Approach 2:
The patent changes the fundamental parameter of graph node identification from using representation-specific identifiers to using canonical identifiers. This parameter change transforms the graph structure from having multiple redundant nodes for the same dataset to having a single node per dataset, with representation information stored as attributes or metadata on the node, thereby reducing complexity while preserving access flexibility.
Data Source
AI summary
In some implementations, a system may receive information identifying a dataset. The system may process an identification attribute using a function that generates a first value, to generate a first identifier for the dataset. The system may search a data store storing a plurality of groupings to identify a grouping with the first identifier for the dataset. The system may extract a second identifier from the grouping with the first identifier for the dataset. The system may search a data lineage based graph representation of a plurality of datasets to identify a graph node representing the dataset. The system may update the data lineage based graph representation of the plurality of datasets to link the dataset with the at least one other dataset based on searching the data lineage based graph representation of the plurality of datasets to identify the graph node.