Cloud Storage File Slicing and Hash-Based Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current cloud storage schemes face issues such as resource waste when uploading large files, as they do not efficiently manage file duplication and storage addressing, leading to potential redundant data uploads and inefficient resource allocation.
Innovation Solution
The proposed solution involves generating unique file and slice IDs using a hash function (like MD5) for file slicing, maintaining relationships between these IDs, and using a consistent hash algorithm to determine storage addresses and detect duplication, thereby optimizing file storage and retrieval processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If traditional cloud storage schemes are used for uploading large files, then files can be stored on servers, but resource waste occurs due to redundant data uploads and inefficient duplication management
Solution Approach 1:
The patent divides large files into smaller slice files with unique slice IDs generated by hash functions. This segmentation enables independent verification and storage of file slices, allowing the system to identify and eliminate duplicates at the slice level rather than uploading entire redundant files, thereby reducing resource waste and improving upload efficiency
Solution Approach 2:
The patent performs preliminary hash computation and duplicate detection before actual file upload. By generating slice IDs and checking for duplicates in advance, the system prevents redundant uploads from occurring, addressing resource waste before it happens rather than after
2Loss of substance
If digest information verification is performed for every backup request, then data duplication can be prevented, but system complexity and processing time increase
Solution Approach 1:
The patent changes the verification parameter from comparing entire file contents to comparing hash values (slice IDs). This parameter transformation maintains the ability to detect duplicates while dramatically reducing the computational complexity and data processing requirements of the verification system
3Measurement precision
If file slicing and hash-based identification are implemented, then duplicate detection accuracy improves, but computational overhead and processing time increase
Solution Approach 1:
The patent uses hash functions to create compact digital representations (slice IDs) of file slices. These hash copies serve as unique identifiers that enable rapid duplicate detection without requiring comparison of the actual file data, thus maintaining high detection accuracy while minimizing processing time
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A server receives a request to upload a file form a user device. The server may generate a unique file ID associated with the file based on the request, and transmit the file ID to the user device. The user device may divide the file to generate multiple slice files and their corresponding slice IDs. From the user device, the server receives the multiple slice files and the slice IDs. Based on the slice IDs, the server may determine storage addresses at which to store the multiple slice files in a distributed storage system using a consistent hash algorithm. The multiple slice files may be stored based on the corresponding storage addresses of the distrusted storage system.