Virtual Copy Splitting in Distributed Database Tablets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current repartitioning methods in distributed databases are inefficient, particularly when splitting large groups with heavy write loads, leading to increased time and resource consumption.
Innovation Solution
Implementing a mechanism that allows for the creation of virtual copies of data within distributed transactions, enabling efficient data splitting by sharing on-disk copies and using a two-phase commit protocol to manage the transfer of data between replicated partitions without duplicating file contents.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large groups are split by copying data to be moved, then data integrity is maintained, but time and resource consumption increase significantly
Solution Approach 1:
The patent uses copy-on-write virtual copies instead of physical data copying. When a group is split, virtual copies of the data are created in the target group that reference the original data files without duplicating them. Only when the original files are modified does the system create actual copies, thus maintaining data integrity while avoiding the time and resource overhead of immediate full duplication.
Solution Approach 2:
The system performs preliminary actions by creating virtual copies and metadata references before actual data movement occurs. The split operation is initiated by creating virtual file system references and updating metadata, allowing the repartitioning to appear complete immediately while the actual data copying is deferred and performed lazily only when needed.
2Productivity
If data is copied to move it during repartitioning, then data can be transferred between groups, but resource consumption increases
Solution Approach 1:
The patent implements virtual copying where target groups receive references to source group data rather than physical copies. This allows data transfer to be represented metadata updates and virtual file system operations, dramatically reducing I/O resources and computational overhead while maintaining the ability to transfer data between groups efficiently.
Solution Approach 2:
The system introduces a virtual file system layer as an intermediary between source and target groups. This virtual layer manages data references and handles the complexity of data movement, allowing efficient data transfer by managing file descriptors and metadata without requiring direct physical data copying between storage systems.
3Quantity of substance
If virtual copies are created without duplicating data, then space requirements are reduced, but system complexity increases
Solution Approach 1:
The patent creates virtual copies that share underlying data storage with the original files. The virtual copy mechanism uses file descriptors, references, and metadata to point to the same physical data blocks, reducing storage space requirements by eliminating redundant data copies while maintaining the ability to access and modify data independently in each group.
Solution Approach 2:
The system implements a nested structure where virtual file systems and metadata layers are nested around the physical data storage. The virtual copy mechanism creates layers of abstraction that reference underlying data, allowing multiple groups to share the same physical data storage while maintaining independent virtual file system views, thus reducing overall storage space requirements.
Data Source
AI summary
Methods and systems for a distributed transaction in a distributed database system are described. One example includes identifying a request to insert a split point in a source group comprising one or more tablet replicas, each tablet including at least a portion of data from a table in the distributed database system, and the split point splitting data in the source group into a first range and a second range different than the first range; in response to the request: sending a list of filenames in the first range of the source group to a first target group comprising one or more tablet replicas; and creating, at the first target group, a virtual copy of files represented by the list of filenames in the first range, the virtual copy making data of the files available, each using a new name, without duplicating the data of the files.