File Upload Optimization via Data Block Hash Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Uploading large files in traditional client/server networks is costly and inefficient due to high time consumption, resource usage, and network bandwidth requirements, especially when uploading similar files or files with large data sizes, and is exacerbated by poor network connectivity and file size limitations.
Innovation Solution
A method where a client identifies and removes data blocks that can be reconstructed at the server using default values or are redundant, and uses a server-stored library of common data blocks to determine which blocks are known and unknown, allowing only unknown blocks to be uploaded, thereby reducing the file upload size and time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the client uploads the entire file in traditional client/server networks, then the server receives complete file data, but the upload process consumes excessive time, network bandwidth, and computing resources
Solution Approach 1:
The patent divides the file into multiple data blocks and processes them individually. The client identifies known data blocks (using hash matching against server library) and unknown data blocks, then uploads only the unknown blocks. This segmentation approach transforms a single large upload operation into multiple smaller, selective uploads, significantly reducing total upload time and network bandwidth consumption.
2Reliability
If the client uploads redundant or reconstructible data blocks, then the server receives complete data, but unnecessary network traffic and server processing overhead increase
Solution Approach 1:
The patent extracts and identifies known data blocks from the file using hash value comparison against the server's library. These known blocks (including reconstructible blocks and default data blocks) are removed from the upload set. Only the remaining unknown data blocks are uploaded to the server, eliminating redundant network traffic while ensuring data completeness through server-side reconstruction of known blocks.
3Productivity
If the server stores a library of common data blocks, then data block reuse and reconstruction efficiency improve, but server storage requirements and library management complexity increase
Solution Approach 1:
The patent implements a server-side library storing common data blocks that are frequently reused across multiple files. The library is organized with hash value indexes for efficient retrieval. When uploading files, the client compares data block hashes against the library to identify known blocks. This local quality approach stores only essential common blocks on the server while maintaining high reconstruction efficiency through hash-based matching.
Data Source
AI summary
The subject disclosure is directed towards technology by which data transmission sizes are reduced when uploading files over a network. By processing hash values corresponding to a plurality of data blocks of a file to potentially be uploaded to a server, the server identifies any already known data block or blocks of the file. The server performs a server-local copy operation that writes the known data block into a server-local copy of the file. If applicable, the server returns hash values corresponding to unknown data blocks to a client, by which the client responds by uploading copies of the unknown data blocks. Accordingly, the client and the server maintain the server-local copy of the file by transferring only unknown data blocks.