Data Structure Management via Frequency-Based Token Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data management systems face challenges in efficiently merging data structures with non-key related data content, particularly when dealing with noisy, incomplete, or inaccurate data, leading to inaccuracies and computational inefficiencies in data structure management.
Innovation Solution
The proposed solution extends the hybrid strict joining and fuzzy matching technique by applying a first filtering layer based on strict matching and generating tokens for data values based on their frequency, using inverse document frequency to identify core data values, and then performing similarity matching to generate potential candidates, thereby maintaining computational performance while achieving high coverage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If current data management solutions are used to merge data structures with non-key related data content, then data structure management can be performed, but the process is time consuming and inaccurate
Solution Approach 1:
The patent segments the data matching process into multiple stages: first filtering layer using strict matching on tokenized data values, second filtering layer using additional strict matching criteria, and finally similarity scoring for remaining candidates. This segmentation allows the system to quickly eliminate non-matching records while applying more computationally intensive similarity analysis only to promising candidates, thereby improving accuracy without excessive time cost.
Solution Approach 2:
The patent performs preliminary actions by tokenizing data values and building inverted indexes before the actual matching process. This preprocessing organizes the data in a way that enables efficient filtering and candidate generation, reducing the computational burden during the actual merge operation and improving both speed and accuracy.
2Quantity of substance
If a large number of data structures with large amount of data content instances are stored, then comprehensive data coverage is achieved, but storage costs and access time increase
Solution Approach 1:
The patent introduces inverted indexes as an intermediary structure that maps data values to the data structures containing them. This intermediary enables efficient location and access of data content instances without requiring linear search through all stored data structures, thereby reducing access time while maintaining comprehensive storage capacity.
3Measurement precision
If strict matching is applied to all data content instances, then matching accuracy is improved, but computational performance deteriorates
Solution Approach 1:
The patent applies partial strict matching by first tokenizing data values and applying strict matching only on tokens, then applying additional strict matching criteria in a second filtering layer, and finally using similarity scoring only for remaining candidates. This partial application of strict matching maintains accuracy for critical fields while reducing computational burden overall.
Solution Approach 2:
The patent changes the parameter of matching strictness dynamically across different filtering stages. The first filtering layer uses strict matching on tokenized values with configurable thresholds, the second filtering layer applies additional strict criteria, and the final similarity scoring adjusts the strictness based on candidate quality. This parameter adjustment maintains accuracy where needed while improving computational performance overall.
Data Source
AI summary
A computing device generates a first token for first data content that is associated with a first relationship and a second relationship, and a second token for second data content that is associated with the first relationship and a third relationship, such that the first token and second token are generated based on a frequency of use of data values included in the first and the second data content. The computing device calculates a first similarity score of data values from third data content that is associated with the second relationship and a fourth relationship with data values from fourth data content that is associated with the third relationship and the fourth relationship in response to the first and second token matching. The computing device then performs, in response to the first similarity score satisfying a similarity threshold, a first modification to any of the data content.