Distributed Database In-Place Resize via Metadata Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed databases face challenges in dynamically responding to changing data requirements and query loads, leading to performance issues and downtime during resizing, as traditional methods involve copying entire data sets, which is time-consuming and disruptive.
Innovation Solution
The method involves resizing a distributed database by adding or removing nodes while maintaining existing nodes, transferring only metadata during the process, and allowing remaining nodes to continue servicing requests immediately, thus minimizing downtime and enabling in-process operations to be queued and resumed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional database resizing methods are used, then database size can be changed, but significant downtime and service interruptions occur
Solution Approach 1:
The patent segments the database into multiple independent shards distributed across compute nodes. During resizing, only the metadata of specific shards needs to be transferred, not the entire database. This segmentation allows the resize operation to proceed while other shards continue to serve requests, dramatically reducing downtime.
Solution Approach 2:
The patent uses a metadata copy approach where only the location information and configuration data of database shards are copied during resize operations, not the actual data. This selective copying enables rapid resizing with minimal disruption to ongoing database services.
2Adaptability or versatility
If entire data sets are copied during resize, then database can be resized, but the process is time-consuming and disruptive
Solution Approach 1:
The patent extracts only the essential metadata information (shard locations, configuration, and indexing data) during resize operations, leaving the actual data in place. This extraction approach reduces the amount of data that needs to be copied from potentially terabytes to mere kilobytes, making resizing fast and non-disruptive.
Solution Approach 2:
The system performs preliminary actions by pre-establishing a metadata store that tracks the location of all database shards before resizing. This pre-configured metadata structure enables rapid reconfiguration during resize operations without needing to scan or copy the actual data, significantly improving resize speed.
3Adaptability or versatility
If nodes are added or removed from distributed database, then database size changes, but existing nodes cannot continue servicing requests
Solution Approach 1:
The patent implements dynamic node management where the database topology can be reconfigured in real-time. When nodes are added or removed, the system dynamically updates the metadata to reflect the new topology, allowing remaining nodes to continue servicing requests without interruption. This dynamic approach maintains service continuity during resizing operations.
Solution Approach 2:
The patent introduces a metadata store as an intermediary that coordinates node additions and removals. This intermediary maintains the mapping between compute nodes and database shards, enabling seamless redistribution of data ownership during resizing while ensuring that existing nodes can continue to serve requests based on the updated metadata.
4Adaptability or versatility
If full data copying is performed during resize, then database structure is updated, but increased loading is placed on computing resources
Solution Approach 1:
The patent copies only metadata (location information, configuration, and indexing data) rather than full data sets during resize operations. This reduces the data transfer volume from potentially terabytes to mere kilobytes, significantly lowering the computational load on resources during the resize process while maintaining database functionality.
Solution Approach 2:
The patent applies partial action by transferring only the essential metadata needed for database reconfiguration, not the complete data sets. This partial copying approach provides just enough information for nodes to redirect their operations during resizing, avoiding the excessive load that would result from copying entire data sets while still achieving the resize objective.
Data Source
AI summary
Techniques for resizing a distributed database are described. A request to resize a distributed database is received. The distributed database stores data organized into one or more rows of one or more tables. Each node of the first plurality of nodes is assigned a portion of the data. A portion of the data assigned to a first node in the first plurality of nodes is selected to be assigned to a second node in a second plurality of nodes. The number of nodes in the first and second plurality of nodes is different, and the first and second plurality of nodes include at least one common node. Metadata of the selected portion of the data is transferred from the first node to the second node. The metadata that includes a location of the selected portion of the data within a provider network.