Database Extent Migration Across Physical Nodes With Dual Metadata Routing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database systems experience increased access latency due to the use of durable storage, which is mitigated by employing physical storage nodes as a cache layer with lower latency NVMe SSDs, and a scalable caching system that allows for dynamic reassignment of virtual groups of extents to physical nodes without downtime.
Innovation Solution
A caching system using physical storage nodes with NVMe SSDs and a scalable caching layer that groups extents into virtual groups, allowing for simultaneous migration and reassignment during changes in physical node availability, with metadata management to support reads and writes during migration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored on durable storage, then data reliability is improved, but access latency increases
Solution Approach 1:
The patent segments the storage system into two distinct layers: a durable storage layer for reliable data persistence and a caching layer using NVMe SSDs for high-speed access. This segmentation allows each layer to fulfill its specific function optimally, resolving the contradiction between reliability and access speed by providing both capabilities through architectural division.
Solution Approach 2:
The NVMe SSD caching layer acts as an intermediary between the database system and the durable storage layer. It mediates data access requests by serving frequently accessed data from the fast cache while maintaining data reliability through the underlying durable storage, thus reducing access latency without sacrificing reliability.
2Adaptability or versatility
If physical nodes are dynamically added or removed, then system scalability is improved, but access downtime occurs
Solution Approach 1:
The system performs preliminary actions by pre-calculating and storing metadata that maps extents to physical nodes before any migration or reassignment occurs. This allows the system to quickly redirect access during node additions or removals without causing downtime, as the routing information is already prepared in advance.
Solution Approach 2:
The patent ensures continuous data access during node transitions by maintaining active caching of extent locations in the metadata layer. This allows the system to seamlessly redirect access to new node configurations without interrupting database operations, achieving scalability without downtime.
3Productivity
If extents are reassigned to different physical nodes, then load balancing is improved, but data migration complexity increases
Solution Approach 1:
The system uses metadata copying as a simplifying mechanism during data migration. Instead of physically moving all data during reassignment, the metadata layer copies and updates extent-to-node mappings, allowing the actual data to remain in place temporarily while access is redirected. This reduces migration complexity while still achieving load balancing goals.
Solution Approach 2:
The metadata layer serves as an intermediary that abstracts the complexity of data migration. It manages extent-to-node mappings and handles redirection logic, isolating the complexity from the actual data movement operations and enabling load balancing without proportionally increasing overall system complexity.
Data Source
AI summary
Techniques are disclosed relating to migrating database extents between physical storage nodes. A database system stores current and new assignment metadata mapping virtual groupings of database extents physical storage nodes, the new assignment metadata reflecting a data migration of extents between the plurality of physical storage nodes. During the migration, the database system receives 1) read requests, which it responds to by reading data from a first physical storage node identified using the current assignment metadata and 2) write requests, which it responds to by writing data to a second physical storage node identified using the new assignment metadata. Upon the migration being complete, the system identifies physical storage nodes accessed by subsequent read and write requests using the new assignment metadata.