Block Map Cache for Distributed Storage Snapshot Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems lack an efficient method for creating and managing snapshots, which are essential for returning a database to a previous state without corrupting critical data, especially in distributed storage and computation environments.
Innovation Solution
The proposed solution involves a network environment with a storage manager that coordinates snapshot creation, quiescing compute nodes to prevent write requests, and storage nodes that finalize segments and manage snapshots, allowing for the creation, cloning, and rollback of storage volumes while maintaining data integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional snapshot creation methods are used in distributed storage systems, then data can be preserved in previous states, but the system complexity and overhead increase significantly
Solution Approach 1:
The patent uses block map caches as simplified copies that track the mapping between logical block addresses and physical storage locations. Instead of copying actual data blocks, the system maintains metadata structures (block maps) that enable rapid snapshot creation by simply copying the block map structure, which dramatically reduces the complexity and overhead of snapshot operations while preserving data integrity.
Solution Approach 2:
The patent extracts the essential snapshot functionality into a separate block map cache structure that is independent from the actual data storage. This separation allows the snapshot mechanism to operate on lightweight metadata rather than heavy data blocks, reducing system complexity while maintaining the ability to restore previous data states.
2Productivity
If block map caches are implemented in memory, then snapshot operations become faster, but memory consumption increases
Solution Approach 1:
The patent segments the block map cache into multiple smaller data structures distributed across the storage system. Instead of maintaining a single large in-memory block map, the system divides the block map into manageable segments that can be stored in memory when needed and persisted to disk when not in use. This segmentation allows fast snapshot operations for active volumes while controlling overall memory consumption through selective caching.
3Reliability
If block maps are persisted to disk, then data durability is improved, but access speed decreases
Solution Approach 1:
The patent implements a dynamic block map cache management system that adapts between in-memory and on-disk storage based on workload requirements. Frequently accessed block maps are loaded into memory for fast access, while less frequently accessed block maps are persisted to disk. The system dynamically loads and evicts block map segments based on access patterns, optimizing the balance between durability and access speed.
Solution Approach 2:
The patent performs preliminary loading of block map segments into memory before they are needed for snapshot operations. By anticipating which block maps will be needed and pre-loading them, the system ensures fast access during critical snapshot operations while maintaining the option to persist data to disk for durability when the system is in a less active state.
Data Source
AI summary
A new snapshot of a storage volume is created by suppressing write requests. Once pending write requests from the computing nodes are completed, storage nodes create a new snapshot for the storage volume by allocating a new segment to the new snapshot. Subsequent write requests to the storage volume are then performed on the segments allocated to the new snapshot. A block map records segments where current data for an LBA of a slice of a storage volume is stored. Block maps may be written to a storage device in order to free memory. Block maps may be read back into memory when needed. Writing and reading of block maps may be performed upon fragments of block maps. On restarting of the storage node, block maps may be restored from block maps stored in the storage device.