Checkpoint Management via Mapping View for Database Re-Sharding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In database systems, tracking checkpoints for shards becomes ineffective during re-sharding operations such as splitting, merging, or moving shards, as the existing checkpoint fails to distinguish between read and unread data, leading to unreliable service.
Innovation Solution
A computer-implemented method and system that generates a mapping view to represent position changes of unprocessed objects across shards, determining an updated checkpoint by mapping source positions to destination positions, ensuring continuous tracking of unread data after shard changes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If re-sharding operations (splitting, merging, moving shards) are performed to manage large data volumes, then data management effectiveness is improved, but checkpoint tracking reliability deteriorates
Solution Approach 1:
The patent introduces a mapping view as an intermediary data structure that tracks the relationship between source shard positions and destination shard positions during re-sharding operations. This mapping view enables the system to translate checkpoint positions from the old shard configuration to the new configuration, thereby maintaining checkpoint tracking reliability despite shard changes. The mapping view acts as a mediator that bridges the old and new shard structures.
Solution Approach 2:
The system performs preliminary actions by pre-calculating and storing the mapping relationships between source and destination shards before re-sharding operations complete. The mapping view is prepared in advance, containing all necessary position translation information, which allows checkpoint updates to be performed efficiently without interruption to data processing operations.
2Device complexity
If existing checkpoint mechanisms are used during re-sharding, then system simplicity is maintained, but service reliability deteriorates due to inability to distinguish read and unread data
Solution Approach 1:
The patent segments the checkpoint tracking function into two distinct components: the original checkpoint mechanism and the new mapping view. The checkpoint continues to record positions in the source shard configuration, while the mapping view separately manages the translation to destination shard configuration. This segmentation allows each component to maintain its simplicity while collectively achieving reliable checkpoint tracking during re-sharding operations.
3Productivity
If shard configurations are changed to optimize data management, then data processing efficiency is improved, but checkpoint accuracy deteriorates
Solution Approach 1:
The system implements a feedback mechanism where the mapping view continuously monitors and records the relationships between source and destination shard positions. When re-sharding operations occur, the mapping view provides feedback information that enables automatic update of checkpoint positions. This feedback loop ensures checkpoint accuracy is maintained even as shard configurations change to optimize data processing efficiency.
Data Source
AI summary
In an embodiment of the present invention, a checkpoint is received, the checkpoint indicating at least one source position of unprocessed object(s) of a plurality of objects on at least one source shard in a database system. In response to detecting a shard change related to the checkpoint to the at least one source shard, the shard change resulting in that the at least one source shard is updated to at least one destination shard, a mapping view is generated for representing position change of the unprocessed object(s) related to the shard change. An updated checkpoint is determined for indicating the at least one destination position of the unprocessed object(s) of the plurality of objects on at least one destination shard in the database system based on the mapping view. With these embodiments, the checkpoint may be managed in a more effective way.