Adaptive File Hashing for Accurate Duplicate Data Assessment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication techniques in the oil and gas upstream industry are inefficient due to high processing overhead and inaccuracies, particularly when dealing with large datasets and files that share similar initial portions or spatially stored data, leading to performance issues and false positives/negatives.
Innovation Solution
A multi-stage adaptive file digest method that uses hash digests from small samples to identify potentially duplicate files, followed by more thorough processing of entire files, reducing unnecessary processing and improving accuracy by excluding unique files early in the process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If hashing of entire contents of all electronic data files is performed to identify duplicates, then measurement precision of duplication is improved, but processing time and computational resources increase significantly
Solution Approach 1:
The patent segments the file assessment process into multiple stages: first hashing a small initial portion of each file to create a preliminary hash digest, then selectively hashing additional portions only for files that appear to be duplicates based on the preliminary assessment. This segmentation reduces the total amount of hashing performed while maintaining accurate duplicate identification.
Solution Approach 2:
The patent applies partial action by performing hashing on only a portion of the file contents initially (the first small portion), and then performing additional hashing only on the portions of files that are suspected to be duplicates. This avoids the excessive action of hashing entire contents of all files upfront, reducing processing time while maintaining accuracy.
2Measurement precision
If one-to-one comparisons of entire data files are conducted to identify duplicates, then duplication identification accuracy is improved, but processing overhead and infrastructure complexity increase
Solution Approach 1:
The patent segments the comparison process by first comparing hash digests of small initial portions of files. Only files with matching preliminary hash digests proceed to more thorough comparison of additional portions. This segmentation dramatically reduces the number of full file comparisons needed while maintaining high accuracy in duplicate detection.
Solution Approach 2:
The patent introduces hash digests as an intermediary mechanism to facilitate duplicate detection. Instead of directly comparing entire file contents, the system first compares compact hash digests generated from small portions of files. This intermediary step simplifies the comparison process and reduces infrastructure complexity while maintaining detection accuracy.
3Measurement precision
If adaptive sampling of file portions is not used and entire files are processed uniformly, then measurement precision is maintained, but processing time and resources are wasted on obviously unique files
Solution Approach 1:
The patent implements a dynamic, adaptive processing approach where the amount of hashing performed on each file is adjusted based on preliminary assessment results. Files that appear to be duplicates receive more thorough processing (hashing of additional portions), while files that are clearly unique receive minimal processing. This dynamic adaptation maintains measurement precision while significantly improving productivity.
Solution Approach 2:
The patent changes the processing parameters adaptively: the size of the file portion to be hashed and the number of hashing iterations performed are adjusted based on the results of preliminary hashing. This parameter adjustment allows the system to maintain high accuracy in duplicate detection while optimizing processing efficiency by avoiding unnecessary computation on unique files.
Data Source
AI summary
Provided in some embodiments are systems and methods for assessing electronic data duplication. Embodiments include extracting first samples of electronic data files and applying a first hash function to the first samples to generate first hash digests. Determining first groupings of the electronic data files having a same file size and first hash digest. Extracting second samples the electronic data files of the first groupings and applying a second hash function to the second samples to generate second hash digests. Determining second groupings the electronic data files having a same file size, a same first hash digest and a same second hash digest. Applying a third hash function to the contents of the electronic data files of the second groupings to generate third hash digests. Determining duplicate electronic data files having a same file size and same first, second, and third hash digests.