Distributed Object Storage Deduplication and Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deduplicated object storage systems face challenges in optimizing storage efficiency and reducing costs while maintaining high performance, particularly in cloud computing environments, where data redundancy and compression techniques do not adequately address the need for reduced storage requirements and efficient data transfer.
Innovation Solution
The method involves storing an uncompressed and non-deduplicated copy of an object on a server, deduplicating and compressing it, and distributing the deduplicated and compressed copies to remote servers, with load balancing to manage copy distribution and prioritize non-compressed copies for reading, thereby reducing storage needs and bandwidth usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is compressed and deduplicated, then storage efficiency is improved, but data transfer and processing performance deteriorates
Solution Approach 1:
The patent segments data into multiple versions: full uncompressed copies for performance-critical operations and compressed/deduplicated copies for storage efficiency. This segmentation allows the system to serve different operational needs from different data representations, resolving the contradiction between storage efficiency and transfer performance.
Solution Approach 2:
The system dynamically selects which data version to use based on operational context - uncompressed versions for read-intensive operations and compressed versions for storage and write operations. This dynamic approach allows the system to optimize for the current operational requirement rather than being locked into a single mode.
2Reliability
If more data copies are distributed across nodes, then system reliability is improved, but storage costs and bandwidth usage increase
Solution Approach 1:
The patent applies different data representation qualities to different locations in the system. Each node maintains local uncompressed copies for immediate access while compressed copies are distributed to alternative nodes. This local quality differentiation allows the system to achieve reliability through distribution while controlling overall storage costs by using compression where appropriate.
Solution Approach 2:
The system creates multiple copies of data but uses compression and deduplication techniques to reduce the actual storage footprint of these copies. By distributing compressed copies rather than full copies, the system maintains reliability through redundancy while significantly reducing the total storage capacity required.
3Quantity of substance
If compression is applied to data, then storage requirements are reduced, but processing speed and read performance deteriorate
Solution Approach 1:
The system performs preliminary decompression of compressed data into uncompressed working copies before processing or reading operations. This preliminary action ensures that when data needs to be accessed or processed, it is already in the optimal format (uncompressed), eliminating the performance penalty that would otherwise be incurred during the actual operation.
Solution Approach 2:
The patent introduces uncompressed working copies as intermediary representations between compressed storage and processing operations. These intermediaries serve as a buffer that allows compressed data to be stored efficiently while providing fast access paths through the uncompressed intermediaries when needed, thus mediating between the conflicting requirements of storage efficiency and processing speed.
Data Source
AI summary
Embodiments for optimized deduplicated object storage system by one or more processors. An uncompressed and non-deduplicated copy of an object may be stored on a node in a distributed object storage system and one or more compressed and deduplicated copies of the uncompressed and non-deduplicated copy of the object may be distributed on one or more alternative nodes in the distributed object storage system.