Dataset Attribute Association via Hash-Based Similarity Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing enterprise software systems face inefficiencies in associating new datasets with existing business classes, particularly when uploading new data, as current methods are resource-consuming and lack effective mechanisms for identifying matching attributes across different data sources.
Innovation Solution
The system employs a hash function-based comparison and similarity analysis to efficiently associate new dataset attributes with existing business classes, using a flow diagram process that includes determining hash values, pruning candidate joins, and calculating similarity values to reduce resource consumption and improve association accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If traditional methods are used to associate new datasets with existing business classes, then association accuracy can be maintained, but resource consumption increases significantly
Solution Approach 1:
The patent segments the attribute association process into multiple stages: first computing hash values for quick filtering, then performing detailed similarity comparisons only on candidates that pass the hash filter. This segmentation reduces resource consumption by avoiding full comparisons for all attribute pairs while maintaining association accuracy through multi-stage verification.
Solution Approach 2:
The patent performs preliminary hash value computation and candidate pruning before executing resource-intensive similarity comparisons. By pre-filtering attribute pairs using hash-based methods and inclusion tests, the system reduces the search space early in the process, thereby conserving resources while preserving the ability to accurately identify matching attributes.
2Loss of time
If comprehensive attribute comparison is performed to ensure accurate association, then association accuracy improves, but processing time increases
Solution Approach 1:
The comparison process is divided into sequential stages: hash value computation, inclusion test, candidate pruning, and detailed similarity comparison. Each stage filters out non-matching attributes early, so that time-consuming precision comparisons are performed only on a small subset of promising candidates, reducing overall processing time while maintaining accuracy.
Solution Approach 2:
The patent applies partial comparison by performing hash-based filtering and inclusion tests on all attributes, then performing complete similarity comparisons only on a pruned subset of candidates. This partial approach to comparison reduces processing time while excessive verification through multiple filtering stages ensures association accuracy is maintained.
3Device complexity
If all attribute pairs are compared to identify matching domains, then association completeness improves, but device complexity increases
Solution Approach 1:
The system performs preliminary hash value computation and inclusion testing for all attribute pairs to identify candidate matches before performing detailed comparisons. This preliminary action ensures that no potential matches are missed (maintaining completeness) while the pruning based on hash mismatches reduces the number of detailed comparisons needed, simplifying the overall system complexity.
Solution Approach 2:
The patent introduces hash values and inclusion tests as intermediary filtering mechanisms between the initial attribute pairs and the final similarity comparison. These intermediaries efficiently eliminate non-matching pairs without requiring complex comparison logic, thereby reducing system complexity while maintaining complete identification of matching domains through multi-stage verification.
Data Source
AI summary
A system comprises reception of a dataset comprising at least one dataset attribute and one or more instances of each dataset attribute, determination of a similarity value between instances of each pair of one of the at least one dataset attribute and one of a plurality of stored attributes, determination of an intersection between instances of each pair of one of the at least one dataset attribute and one of the stored attributes having a similarity greater than a threshold, and generation of metadata associating a dataset attribute with a semantic entity with which a stored attribute having a similarity to the dataset attribute greater than the threshold is associated.