Graph Duplicate Identification Using Version Comparison Scores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for duplicate determination in graph-based data systems, such as master data management systems, face challenges in accurately identifying and merging duplicate records across different versions of a graph, especially when changes occur over time, leading to inefficiencies in data deduplication processes.
Innovation Solution
A method for duplicate identification in graphs that compares nodes and subgraphs across different versions of a graph, using a comparison score that considers both current and historical attributes and relationships, with a penalty weight to account for changes, to determine the similarity and potential duplication of entities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If only the current version of the graph is used for duplicate determination, then the processing speed is faster, but the accuracy of identifying duplicates that changed over time deteriorates
Solution Approach 1:
The system pre-computes and stores comparison scores between nodes across different graph versions before duplicate determination is needed. These pre-computed scores are stored in a cache or database, allowing the system to quickly retrieve historical similarity information without performing expensive graph comparisons in real-time, thus resolving the contradiction between accuracy and processing time
Solution Approach 2:
The system dynamically adjusts which graph versions are compared based on the specific duplicate determination query. Instead of always comparing all historical versions, the system intelligently selects relevant versions to compare, adapting the comparison scope to the specific needs of each query while maintaining accuracy for changed entities
2Measurement precision
If multiple previous versions of the graph are compared, then the accuracy of duplicate identification improves, but the computational complexity increases
Solution Approach 1:
Comparison scores between nodes across different graph versions are pre-computed and stored in advance. This preliminary computation eliminates the need to perform expensive graph traversal and node attribute comparisons when duplicate determination is actually needed, significantly reducing computational complexity while maintaining high accuracy
Solution Approach 2:
The system applies different comparison strategies to different parts of the graph based on their change history. Nodes that have changed across versions receive more thorough multi-version comparison, while stable nodes use simpler comparison methods, optimizing computational resources according to local needs
3Measurement precision
If historical graph versions are considered in duplicate determination, then the matching accuracy of entities that changed over time improves, but the data processing overhead increases
Solution Approach 1:
The system pre-processes historical graph versions by computing and storing comparison scores between nodes, creating a ready-to-use reference database. This preliminary action transforms complex historical data into compact, query-friendly formats, enabling fast retrieval during duplicate determination without the overhead of processing raw historical versions in real-time
Solution Approach 2:
Instead of storing and processing complete historical graph versions, the system creates and stores simplified copies in the form of comparison scores and similarity metrics. These compact representations capture the essential information needed for duplicate determination while occupying minimal storage space and enabling rapid processing
Data Source
AI summary
Duplicating identification in a graph having a second version. The graph includes nodes representing entities and edges representing relationships between the entities. The method includes: providing a previous first version of the graph. At least two target nodes of the graph may be identified. The first version and the second version of the graph may be compared for determining a comparison score indicative of a similarity between the two target nodes. The comparison score may be used for determining whether the two target nodes are duplicates with respect to each other.