Cloud Storage Deduplication via Time-Series Metadata
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud-based storage systems face challenges in managing data consistency, redundancy, and performance, especially with hard drive failures, leading to substantial management overhead and increased costs due to premium pricing for additional storage capacity.
Innovation Solution
A distributed cloud data management system (DCDMS) is introduced, which leverages a distributed time-series database across multiple compute nodes to track object requests and provide snapshot, clone, and roll-back operations, deduplicating data to reduce storage costs and enhance scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If cloud-based storage systems are used to provide flexible and scalable storage capacity, then storage adaptability and scalability are improved, but storage costs increase due to premium pricing for additional capacity
Solution Approach 1:
The patent uses copy-on-write technology to create virtual copies of storage data. When data is cloned or snapshotted, only the metadata is copied immediately while the actual data blocks are shared between original and copy until modified. This allows multiple versions to exist without duplicating storage costs, resolving the contradiction between storage scalability and storage costs.
Solution Approach 2:
The system merges multiple storage operations (cloning, snapshotting, versioning) into a unified storage management framework that operates at the block level. By combining these functions and managing them through a common metadata system, the patent achieves flexible storage adaptability while optimizing storage resource utilization and reducing costs.
2Reliability
If data redundancy and consistency are maintained across distributed storage devices, then data reliability is improved, but management overhead increases substantially
Solution Approach 1:
The patent introduces a metadata service as an intermediary layer between client applications and the distributed storage system. This metadata service manages data location, versioning, and consistency information centrally, while the actual storage operations are distributed. This intermediary approach maintains data reliability and consistency without requiring complex coordination across all storage devices, thus reducing management overhead.
Solution Approach 2:
The system segments storage management into two independent layers: metadata management (handled by the metadata service) and data storage (handled by distributed storage devices). This segmentation allows consistency and redundancy to be managed at the metadata level without complicating the underlying storage operations, reducing overall management overhead while maintaining reliability.
3Adaptability or versatility
If additional storage capacity is provisioned to meet sudden business needs, then storage adaptability is improved, but storage costs increase substantially
Solution Approach 1:
The patent enables rapid storage expansion through cloning and snapshot operations that create instant virtual copies of existing data without requiring physical duplication of storage capacity. Organizations can flexibly provision additional storage views and versions on-demand, paying only for the metadata overhead rather than full data duplication, thus maintaining storage flexibility while controlling costs.
Solution Approach 2:
The system performs preliminary actions by maintaining multiple versions and snapshots of data in advance. When business needs suddenly increase, organizations can immediately activate pre-existing snapshots or clones to meet demand without provisioning new storage capacity, thereby achieving storage flexibility while avoiding additional storage costs.
Data Source
AI summary
The disclosed embodiments disclose techniques for managing cloud-based storage using a time-series database. A distributed cloud data management system (DCDMS) manages objects stored in a cloud storage system. The DCDMS leverages a distributed time-series database to track objects accessed via the DCDMS. During operation, the DCDMS receives a request to access an object using a path identifier and an object identifier. The DCDMS determines from the path identifier that the request is associated with one of its supported extended capabilities, and uses the previously tracked object operations that are stored in the time-series database to determine the actual target bucket in the cloud storage system that contains the requested object; the target bucket that contains the object may be different from the bucket identified in the path identifier that is received. The object identifier is then used to access the requested object from the target bucket to service the request.