Digest-Based Block Storage for Snapshot De-Duplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Multi-tiered data storage systems face challenges in maintaining system operability, scalability, and dependability while avoiding unnecessary data duplication and ensuring the ability to revert to a desired system state without impacting other users or compromising integrity.
Innovation Solution
A digest-based block storage engine is introduced, which generates digests using strong hash functions for data blocks, manages reference counts, and takes snapshots across multiple time axes, enabling independent modifications and access to past system states while maintaining system integrity and avoiding duplication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If de-duplication is implemented to avoid unnecessary data duplication, then storage efficiency is improved, but system complexity increases due to the need to manage reference counts and digest verification
Solution Approach 1:
A digest-based block storage engine is introduced as an intermediary layer between the file system and storage unit. This engine generates digests (hash values) for data blocks and manages reference counts, automatically handling de-duplication operations without requiring complex application-level implementation. The intermediary engine abstracts the complexity of de-duplication management while improving storage efficiency.
Solution Approach 2:
Instead of storing multiple copies of identical data blocks, the system stores a single copy and uses digest-based identification to recognize duplicates. When the same data is encountered, the system references the existing block through its digest, avoiding redundant storage while maintaining data accessibility through reference counting mechanisms.
2Reliability
If writable snapshots are implemented to enable reverting to desired system states, then system dependability is improved, but data duplication increases as multiple instances share blocks
Solution Approach 1:
Multiple writable snapshots share common data blocks through the digest-based storage engine. When snapshots reference the same data, they share the underlying storage blocks rather than duplicating them. The engine manages reference counts to ensure data integrity while allowing multiple snapshots to coexist efficiently, reducing overall data duplication while maintaining the ability to revert to any snapshot state.
3Adaptability or versatility
If multi-tiered storage is implemented to increase storage capacity, then storage scalability is improved, but system operability deteriorates when data is shared among objects in different tiers
Solution Approach 1:
The digest-based block storage engine provides a universal interface that works consistently across multiple storage tiers. By identifying data blocks through digests rather than physical locations, the system enables data to be shared and moved between different storage tiers (hot, warm, cold storage) without compromising operability. The engine abstracts tier-specific details, providing uniform data access and management across the multi-tiered architecture.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The solution enhances system scalability and dependability by allowing independent modifications across multiple universes, reducing data duplication, and ensuring seamless access to past system states without impacting other users, thus improving overall system operability and integrity.
Implementation Method 1
the digest is based on a value calculated by applying a strong hash function to a data block
Data Source
AI summary
A system comprising a file system that manages files and directories. A digest-based block storage engine coupled to the file system generates a digest when performing saving and retrieving operations initiated by the file system. The engine counts references to data blocks and a list of free data blocks. A storage unit may be coupled to the engine, wherein the storage unit stores a plurality of data blocks.