Dataset Comparison With Field-Subset Hash Signatures
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data comparison techniques are inefficient and prone to errors, especially when dealing with large datasets containing irrelevant fields, and fail to distinguish between content and non-content fields, leading to mismatches when the content is the same.
Innovation Solution
Computing hash values over a selected subset of fields to generate a signature for each dataset, which is then compared to determine matching content, while disregarding irrelevant fields, and optionally insensitive to record order.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If detailed or manually-assisted comparison of data fields is performed, then comparison accuracy is improved, but time consumption and labor intensity increase significantly
Solution Approach 1:
The patent segments the data comparison task by dividing it into two stages: first computing hash values for each data record to enable quick filtering, then performing detailed field-by-field comparison only on records with matching hash values. This segmentation resolves the contradiction by making the comparison process efficient while maintaining accuracy through selective detailed verification.
Solution Approach 2:
The patent introduces hash values as an intermediary element that mediates between the raw data fields and the final comparison result. By computing hash values from data fields and comparing these intermediaries first, the system achieves fast preliminary filtering, with detailed comparison reserved only for promising candidates, thus balancing speed and accuracy.
2Productivity
If bulk comparison techniques are used, then processing speed is improved, but the ability to distinguish between content fields and non-content fields is lost, leading to false mismatches
Solution Approach 1:
The patent segments the data fields into content fields and non-content fields, and applies different processing strategies to each segment. Content fields are included in hash value computation for meaningful comparison, while non-content fields are excluded or handled separately. This segmentation enables both fast bulk processing and accurate field distinction.
Solution Approach 2:
The patent applies local quality by treating different fields with different levels of importance. Content fields that affect data substance are given special treatment through selective inclusion in hash computation, while non-content fields are handled differently. This local differentiation maintains processing speed while ensuring accurate distinction between meaningful and meaningless variations.
3Productivity
If hash values are computed over all fields, then comparison speed is improved, but storage requirements and computational overhead increase
Solution Approach 1:
The patent extracts only the essential content fields from the complete data record to compute hash values, excluding non-content fields and metadata. This extraction reduces the amount of data processed and stored while maintaining the ability to detect meaningful differences, thus improving speed and reducing storage requirements simultaneously.
Solution Approach 2:
The patent applies partial action by computing hash values over a subset of fields rather than all fields. This partial computation is sufficient for the intended purpose of detecting data substance changes, while avoiding the unnecessary computational and storage overhead of processing every single field in the dataset.
Data Source
AI summary
Relevant content of datasets is compared by computing a hash value over a selected subset of fields to obtain a signature of a dataset, with other fields being disregarded. A hash value can be computed directly for all records of the dataset, or by combining individual hash values for each record. Comparison of the signature with that of other datasets leads to efficient determination whether two datasets match with respect to relevant content in the selected fields. For larger groups of datasets, lists of matched and mismatched datasets can be reported. Optional features include matches insensitive to permutation of the records, or identification of which records in a dataset fail to match.