Database Extent Mapping with Virtual Groups for Scalable Node Reassignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database systems experience higher access latency due to the use of durable storage, which has hardware limitations compared to other storage technologies, leading to performance issues in certain applications.
Innovation Solution
Implement a scalable caching layer using physical storage nodes equipped with high-performance devices like NVMe SSDs, grouping database extents into virtual groups to facilitate dynamic reassignment during changes in the number of physical nodes, and maintaining multiple versions of assignment metadata to manage data migration with minimal downtime.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If durable storage is used for database extent storage, then data reliability is improved, but access latency increases due to hardware limitations
Solution Approach 1:
The patent segments the storage system into two distinct layers: a durable storage layer for long-term data retention and a caching layer using physical storage nodes with high-performance devices (e.g., NVMe SSDs) for fast access. This segmentation allows the system to simultaneously achieve data reliability through durable storage and low access latency through the caching layer, as the caching layer handles frequent read operations while the durable storage layer maintains data persistence.
Solution Approach 2:
The patent introduces a caching layer as an intermediary between the database system and the durable storage layer. This caching layer, implemented using physical storage nodes with high-performance storage devices, mediates access requests by serving frequently accessed extents from fast storage while allowing less frequently accessed extents to reside in durable storage, thereby reducing overall access latency without compromising data reliability.
2Adaptability or versatility
If physical storage nodes are added or removed to improve scalability, then system adaptability is improved, but data migration complexity increases
Solution Approach 1:
The patent implements self-service mechanisms where the system automatically manages data migration when physical storage nodes are added or removed. The metadata service automatically updates extent-to-node mappings, and the system handles data redistribution without requiring manual intervention, thereby reducing migration complexity while maintaining scalability.
Solution Approach 2:
The patent employs dynamic extent-to-node mapping where the assignment of extents to physical storage nodes is not fixed but can be dynamically adjusted based on system conditions. When storage nodes are added or removed, the system dynamically remaps extents to available nodes, enabling seamless scalability. This dynamic approach is facilitated by storing mapping information in a metadata service that can be updated without system downtime.
3Productivity
If virtual groups of extents are used to manage storage allocation, then storage management efficiency is improved, but metadata storage requirements increase
Solution Approach 1:
The patent merges multiple extents into virtual groups, where each virtual group is assigned to a specific physical storage node. This merging approach improves storage management efficiency by allowing the system to manage extents in aggregated units rather than individually, reducing the overhead of tracking and allocating individual extents. Although this creates additional metadata structures, the virtual group concept simplifies the overall management architecture by providing a hierarchical organization level.
Data Source
AI summary
Techniques are disclosed relating to storing database extents in physical storage nodes. To store the extents, a physical storage node of a computer system first accesses assignment metadata, which includes determining 1) virtual groupings of database extents assigned to the physical storage node and 2) database extents associated with the determined one or more virtual groupings. For a given database extent, a corresponding virtual grouping is determinable by performing a first hashing operation that uses an identifier for the given database extent. The physical storage node then accesses and stores the determined database extents. The physical storage node can now service requests for data of the database system that are stored at the first physical storage node.