Cloud Storage Deduplication with Chunked Batch Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The inefficient use of cloud storage due to duplicate data copies and increased costs associated with managing data in cloud storage systems, particularly when data is copied from a datacenter to cloud storage, leading to suboptimal storage and operational expenses.
Innovation Solution
Implementing a data plane with deduplication techniques, chunking data into variable sizes, and using batch processing to efficiently store data in cloud storage by identifying and eliminating duplicates, optimizing batch sizes, and managing encryption keys to ensure secure and efficient data management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data is copied from datacenter to cloud storage, then data accessibility is improved, but storage efficiency deteriorates due to duplicate copies
Solution Approach 1:
The patent segments data into fixed-size chunks before uploading to cloud storage. Each chunk is independently hashed and stored, allowing the system to identify and eliminate duplicates at the chunk level rather than storing entire duplicate files, thereby improving storage efficiency while maintaining data accessibility.
Solution Approach 2:
The patent uses cryptographic hashing to create unique identifiers for each data chunk. By comparing hashes of local data chunks with those already stored in cloud storage, the system identifies duplicates and avoids storing redundant copies, thus improving storage efficiency without compromising the ability to retrieve data when needed.
2Reliability
If all data is uploaded to cloud storage, then data security is improved through centralized management, but operational costs increase
Solution Approach 1:
The patent performs preliminary deduplication operations locally before uploading data to cloud storage. By computing hashes of data chunks and comparing them with existing cloud storage hashes in advance, the system identifies duplicates and excludes them from upload, reducing the volume of data transferred and stored, thereby lowering operational costs while maintaining centralized security management.
Solution Approach 2:
The patent changes the parameter of data representation by storing only unique data chunks and their metadata in cloud storage rather than complete duplicate files. This parameter change from storing full copies to storing unique segments significantly reduces storage costs and operational expenses while maintaining data security through centralized cloud management.
3Quantity of substance
If deduplication is performed on all data chunks, then storage efficiency is improved, but processing time increases
Solution Approach 1:
The patent segments data into fixed-size chunks and processes them independently for deduplication. This segmentation allows parallel processing of multiple chunks simultaneously, reducing overall processing time while maintaining high storage efficiency through comprehensive deduplication at the chunk level.
Solution Approach 2:
The patent implements practical optimizations in the deduplication process by using fixed-size chunking and efficient hashing algorithms. Rather than performing exhaustive analysis on entire files, the system applies partial action at the chunk level with optimized processing, achieving sufficient deduplication efficiency without excessive processing time overhead.
Data Source
AI summary
A specification of content to be stored in a cloud storage is received at a client-side component. A first portion of the content is divided into a plurality of data chunks. One or more data chunks of the plurality of data chunks that are to be sent via a network to be stored in the cloud storage are identified. It is determined whether a batch size of the one or more identified data chunks does not meets a threshold size. One or more data chunks of a second portion of the content that are to be stored in the cloud storage are identified. It is determined that a size of a second batch of data chunks that includes the one or more identified data chunks of the first portion of the content and the one or more identified data chunks of the second portion of the content does not meet the threshold size. It is determined that a batch period is greater than or equal to a batch threshold period. The second batch of data chunks is written to a storage of a cloud server included in a data plane.