Hierarchical Duplicate Detection in Document Repositories
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automated techniques for detecting duplicate entities in repositories fail to account for hierarchical relationships, leading to inaccurate duplicate detection and misclassification of entities.
Innovation Solution
An automated method that generates vector representations of entities and applies similarity thresholds specific to their hierarchical relationships, using a machine learning model that can be retrained based on user feedback to improve accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing automated techniques are used to detect duplicate entities, then the detection process is efficient, but the accuracy is poor due to failure to account for hierarchical relationships
Solution Approach 1:
The patent applies local quality by implementing relationship-specific similarity thresholds tailored to different hierarchical relationships. Instead of using a single global threshold, the system defines distinct thresholds for sibling entities, parent-child entities, and ancestor-descendant entities. This allows the duplicate detection to adapt to the specific local characteristics of each relationship type, improving accuracy while maintaining manageable system complexity through structured threshold management.
Solution Approach 2:
The patent implements dynamics by making the similarity threshold adaptive rather than static. The system dynamically selects appropriate thresholds based on the hierarchical relationship between entity pairs being compared. This dynamic adaptation allows the detection system to respond to different contextual requirements of various relationship types, resolving the contradiction between accuracy and complexity.
2Measurement precision
If manual duplicate detection is used, then the accuracy is high, but the process is impractical for large repositories
Solution Approach 1:
The patent applies self-service by enabling the system to automatically perform duplicate detection with relationship-aware thresholds without requiring manual intervention for each entity pair. The automated system selectively applies appropriate thresholds based on hierarchical relationships, achieving both high accuracy and scalability for large repositories, thus resolving the contradiction between manual accuracy and automated efficiency.
3Measurement precision
If a single similarity threshold is used for all entities, then the system is simple to implement, but it produces inaccurate results by misclassifying entities at different hierarchical levels
Solution Approach 1:
The patent implements local quality by defining distinct similarity thresholds for different hierarchical relationship types (sibling, parent-child, ancestor-descendant). This allows the system to account for the varying expectations of similarity at different hierarchical levels, improving detection accuracy while keeping threshold management organized and systematic rather than chaotic.
Solution Approach 2:
The patent applies segmentation by dividing the threshold management into distinct segments corresponding to different relationship types. Instead of managing a single monolithic threshold, the system segments thresholds into relationship-specific categories, making the complexity manageable and the application of appropriate thresholds systematic and error-resistant.
Data Source
AI summary
Aspects of the present disclosure relate to automated detection of duplicate entities in a repository. Embodiments include generating a vector representation of a given entity; comparing the vector representation of the given entity to vector representations of a plurality of entities within a repository to determine a level of similarity for the given entity relative to each of the plurality of entities; selecting a similarity threshold of a set of similarity thresholds for the given entity and a particular entity based on a hierarchical relationship between the given entity and the particular entity, wherein the set of similarity thresholds comprises a sibling threshold, a parent-child threshold, and an ancestor threshold; and determining that the given entity is a duplicate of the particular entity based on a level of similarity between the given entity and the particular entity exceeding the selected similarity threshold.