Automated Database Resharding With Threshold-Based Downtime Control

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Sharded databases often become imbalanced over time, leading to inefficient access and performance issues due to changes in data popularity, making manual resharding costly, time-consuming, and risky, with existing methods requiring significant downtime and human effort.

Innovation Solution

A computer-implemented method and system for automatically resharding a sharded database by determining if the duration of unavailability meets a predetermined threshold, allowing for seamless transition to a new shard key without interrupting read/write operations, using a processor to manage the resharding process and minimize downtime.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If manual resharding is performed to reorganize data according to a new shard key, then database balance and access efficiency are improved, but significant downtime and human effort are required

Engineering Contradiction:

Improvedatabase access efficiencyVSAvoiddowntime during resharding

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by creating a new sharded collection with the desired shard key before the old collection is dropped. It establishes the new collection structure, indexes, and permissions in advance, then switches the application to use the new collection, minimizing the actual downtime to only what's needed for the switchover.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system uses an intermediary approach by maintaining both the old and new sharded collections simultaneously during the transition period. The new collection serves as an intermediary structure that can be prepared without affecting the old collection's operation, allowing for a smooth transition with minimal service interruption.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Stability of the object's composition

If manual resharding is performed to correct imbalanced shard distribution, then data organization is improved, but operational risk and cost increase

Engineering Contradiction:

Improveshard distribution balanceVSAvoidoperational risk during resharding

Core Design Contradiction:

Stability of the object's compositionVSReliability

Solution Approach 1:

The system creates a copy of the existing sharded collection with a new shard key configuration. The new collection is built as a copy with the desired structure, allowing verification and testing before switching. This copying approach reduces operational risk because the original collection remains intact and can be restored if needed.

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The system prepares backup and rollback mechanisms beforehand. It maintains the original collection as a safety cushion, allowing the operation to be reversed if problems occur. This prior cushioning reduces operational risk by ensuring that failures during resharding do not result in data loss or prolonged downtime.

Inventive Principle:

Principle #11Beforehand cushioning (Prior cushioning)

3Speed

If the shard key is changed to improve data distribution, then access performance is improved, but the process requires significant human effort and complexity

Engineering Contradiction:

Improvedata access speedVSAvoidresharding process complexity

Core Design Contradiction:

SpeedVSDevice complexity

Solution Approach 1:

The system performs resharding operations automatically without requiring manual intervention for complex tasks. It self-manages the creation of new collections, copying of data, index building, and permission updates. This automation reduces the complexity burden on operators while still achieving the performance improvements from optimized shard key configuration.

Inventive Principle:

Principle #25Self-service

4Ease of operation

If automatic resharding is implemented to minimize downtime, then database availability is maintained, but automated control and monitoring complexity increases

Engineering Contradiction:

Improvedatabase availability during transitionVSAvoidautomated resharding control

Core Design Contradiction:

Ease of operationVSExtent of automation

Solution Approach 1:

The system implements feedback mechanisms to monitor the resharding process and database status. It tracks collection creation progress, data copying status, and performance metrics to determine when the transition is complete and safe. This automated feedback control maintains database availability while managing the complexity of the automation process through systematic monitoring and decision-making.

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS11893042B2Methods and systems for automatically resharding a sharded collection of data

Publication Date: 2024.02.06 MONGODB INC

AI summary

A method is provided for resharding a sharded database sharded according to a first shard key. The method includes: receiving, by a processor an instruction to reshard the sharded database; receiving, at the processor, a new shard key to be used in a resharding process to reshard the sharded database; determining, by the processor, whether a duration of unavailability of the sharded database during the resharding process is less than a predetermined amount of time; and automatically performing, by the processor, the resharding process according to the new shard key to produce a resharded database, if the duration of unavailability is less than the predetermined amount of time. The method may be performed without users noticing a significant interruption to read/write operations from/to the database.