Large File Hashing with Partial Data for Fast Identity Verification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Hashing large data files to verify file identity is resource-intensive and inefficient, especially in network environments where processing and network resources are limited.
Innovation Solution
A method involving an endpoint device that determines the size of a received file, searches for a record in a data structure, and hashes specific data portions to match previous hash results, allowing verification of file identity without processing the entire file, and creating a file reference for further processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire data of a large file is hashed to verify file identity, then the verification is complete and accurate, but the processing and network resources consumed are excessive
Solution Approach 1:
The patent divides the file verification process into multiple segments by hashing only specific data portions (e.g., headers, footers, or selected blocks) rather than the entire file. The data structure stores records with locations of these particular data portions and their corresponding hash values, enabling partial verification that consumes fewer resources while maintaining security.
Solution Approach 2:
The patent applies partial action by hashing only a subset of the file data (less than the entire file) to verify file identity. The system determines that hashing specific portions (such as 1-10% of the file) is sufficient for security verification, avoiding the excessive resource consumption of full-file hashing while maintaining adequate reliability.
2Reliability
If the entire data of a large file is hashed to verify file identity, then the verification is complete and accurate, but the time required for verification increases
Solution Approach 1:
The verification process is segmented into hashing only specific data portions stored in the data structure records rather than processing the entire file sequentially. This segmentation enables parallel processing of multiple portions and significantly reduces verification time while maintaining security through the distributed nature of the hashed portions.
Solution Approach 2:
The system performs preliminary action by pre-computing and storing hash values of specific data portions in a data structure when the file is first received or created. This allows rapid verification by comparing new hash computations against pre-stored values, eliminating the need to re-hash entire files during each verification operation.
3Use of energy by moving object
If only a portion of the file is hashed for verification, then processing resources are saved, but the verification may be less reliable
Solution Approach 1:
The patent applies local quality by selecting specific data portions (headers, footers, metadata blocks) that have disproportionate importance for file identity verification. These particular portions contain critical information that, when hashed, provide high-value verification with minimal resource consumption, making the partial hashing approach as reliable as full hashing.
Solution Approach 2:
The patent introduces a data structure as an intermediary that stores records linking specific data portion locations to their hash values. This intermediary enables efficient lookup and comparison operations, allowing the system to verify file identity by checking only the stored portions while maintaining reliability through the structured organization of verification data.
4Reliability
If traditional hashing methods are used for large files, then complete file verification is achieved, but network bandwidth is consumed transmitting large hash data
Solution Approach 1:
The patent extracts only the essential verification information (hash values of specific data portions) from the complete file data and stores them in a compact data structure. This extraction eliminates the need to transmit or process the entire file data for verification purposes, significantly reducing network bandwidth consumption while maintaining verification reliability through the extracted hash values.
Data Source
AI summary
Secure hashing of large files to verify file identity. In some implementations, a method includes determining a size of a particular file received by an endpoint device, and searching for a record indexed in a data structure based on the size. In response to finding the record, a sequence of multiple records is accessed in the data structure. For each record of the sequence, a particular data portion is hashed that has a location in the particular file that corresponds to a location in the record to obtain a particular hash result. In response to the particular hash result matching a corresponding previous hash result stored in the record based on an associated data portion in an associated file, the particular file is determined to be the same as the associated file, and characteristics of the particular file are determined using file information for the associated file.