Parallel Hash Calculation for Archive Files
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Calculating hashes of archive files and their file entries is a CPU-intensive process that increases server processing time and resource consumption, especially when files are read multiple times, leading to reduced productivity and increased resource usage.
Innovation Solution
Performing parallel hash calculation of archive files and their file entries in a separate thread during local storage, eliminating the need for additional read operations and reducing total processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If hash calculation is performed sequentially after file storage, then hash values can be accurately calculated, but total processing time increases and server productivity decreases
Solution Approach 1:
The patent applies preliminary action by initiating hash calculation during the file storage process itself, rather than waiting for storage to complete. The hash calculation thread is started concurrently with the storage thread, allowing hash computation to begin before the file is fully written to disk. This overlapping of operations reduces total processing time while maintaining hash calculation accuracy.
Solution Approach 2:
The patent segments the file processing into independent concurrent threads: a storage thread that writes files to disk and a hash calculation thread that computes hash values. By dividing the monolithic sequential process into separate parallel threads, the system achieves both accurate hash calculation and improved server productivity through concurrent execution.
2Reliability
If hash calculation is performed on archive files and all file entries, then complete validation is achieved, but CPU resource consumption increases
Solution Approach 1:
The patent applies local quality by selectively calculating hashes only for specific file entries within archive files based on validation needs, rather than uniformly processing all entries. The system can choose to validate only certain files or use different validation strategies for different archive entries, optimizing CPU resource consumption while maintaining necessary reliability for file transfer validation.
3Measurement precision
If multiple read operations are performed on archive files for hash calculation, then accurate hashes are obtained, but resource usage increases
Solution Approach 1:
The patent merges the file reading operations by having the hash calculation thread read from the same file stream that is being stored, rather than performing separate read operations. The storage thread and hash calculation thread share the incoming file data stream, eliminating redundant I/O operations while ensuring accurate hash calculation from the actual file contents.
Data Source
AI summary
The present disclosure involves systems, software, and computer implemented methods for calculating hashes of archive files and their file entries. In one example, a method may include receiving an archive file which includes at least one compressed file, storing the archive file by sending a first copy of the archive file stream to a first processing thread which stores the archive file in a local file system, and applying a hash function to the archive file and the at least one compressed file to create hash values corresponding to the archive file and the at least one compressed file by sending a second copy of the archive file stream to a second processing thread, which applies the hash function to the archive file and the at least one compressed file to create the hash values and is executed in parallel to the first processing thread.