Shard Management for Scalable Database Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems face challenges in scaling processing capacity to match workload demands, leading to disruptions and high costs due to the need for frequent capacity adjustments, especially in managed relational database services where client applications may experience interruptions from dropped connections.
Innovation Solution
A database system that supports both client-managed and system-managed tables within a common database, allowing for intelligent query routing and dynamic resource allocation, enabling flexible scaling of computing resources to optimize performance without interrupting client applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If database capacity is increased to match peak workload demands, then processing capacity is improved, but cost increases and disruption risk increases during scaling operations
Solution Approach 1:
The database table is divided into multiple shards that can be independently scaled. Each shard is a partition of the table data that can be managed separately, allowing the system to scale only the portion needed for peak workloads without affecting the entire database, thus avoiding application disruptions.
Solution Approach 2:
The system dynamically assigns and reassigned shards to different database nodes based on workload demands. Shards can be moved between nodes without interrupting client connections, enabling flexible capacity adjustment while maintaining continuous service availability.
2Adaptability or versatility
If database capacity is frequently adjusted to match workload changes, then adaptability to workload is improved, but disruption to client applications increases
Solution Approach 1:
By segmenting the table into shards, the system can adjust capacity at the shard level without affecting overall database connectivity. Client connections remain stable while individual shards can be scaled independently to match varying workload patterns.
Solution Approach 2:
The system uses a router as an intermediary between client applications and database nodes. The router manages connection routing and can redirect queries to different shards or nodes during scaling operations, preventing direct disruption to client applications while enabling capacity adjustments.
3Ease of operation
If system-managed tables use automated scaling, then operational complexity is reduced, but control over resource allocation is lost
Solution Approach 1:
The system provides dynamic resource allocation where computing and storage resources are automatically adjusted based on workload. The router dynamically assigns shards to available nodes, enabling flexible resource management that adapts to changing demands while simplifying operational tasks for users.
Data Source
AI summary
Shard management may be implemented for scalable database tables. A table may be identified as aligned with another table of the database according to a common shard key across multiple shards that are mapped to respective storage volumes of a data store of the database according to a placement hierarchy. Different portions of the table may be stored across the shards as respective table slices with corresponding portions of the other table as respective other table slices with same values of the shard key. Metadata for performing access requests to the database may be updated to identify individual ones of the table slices stored with individual ones of the other table slices as respective table slice groups that are mapped to respective ones of the shards in the placement hierarchy.