Columnar Table Merge Availability via Shadow Delta Parts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The merge process in columnar databases, which involves combining delta and main data parts, often results in significant downtime due to reorganization, restricting access to data and delaying query responsiveness.
Innovation Solution
Implementing a second delta part and a shadow second delta part, linked to form a new delta, allows data manipulation during the merge operation while uncommitted rows are copied, reducing the need for exclusive locks and enabling continuous access by using lock-free structures for atomic operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If a merge process is executed to combine delta part and main part of a columnar table, then data compression is improved, but data access availability deteriorates due to exclusive locks
Solution Approach 1:
The columnar table is segmented into multiple delta parts (first delta part, second delta part) and main parts (first main part, second main part). During merge operations, different segments can be accessed independently, allowing data manipulation to continue on unmerged segments while merges occur on other segments, thus maintaining data access availability while achieving compression improvements.
Solution Approach 2:
A shadow second delta part is introduced as an intermediary structure during the merge process. The shadow part receives data manipulations during the merge of the first delta part and main part, acting as a buffer that allows continuous data access without requiring exclusive locks on the entire table, thereby resolving the contradiction between compression and availability.
2Reliability
If exclusive locks are used during merge operation to ensure data consistency, then data integrity is maintained, but query responsiveness deteriorates due to access restrictions
Solution Approach 1:
The system dynamically manages locks and data part activations based on operation type. Read operations can access uncommitted rows in the first delta part without exclusive locks, while write operations during merge are directed to the shadow second delta part. This dynamic approach maintains data integrity while allowing concurrent query access, improving responsiveness without sacrificing integrity.
Solution Approach 2:
The shadow second delta part is created as a copy of the first delta part structure. Data manipulations are copied to the shadow part during merge operations, allowing the original first delta part to remain accessible for queries. This copying mechanism maintains data integrity through the shadow copy while preserving query responsiveness via the original copy.
3Ease of operation
If the first delta part is used for data manipulation during merge, then data manipulation capability is maintained, but access conflicts increase due to lock requirements
Solution Approach 1:
The shadow second delta part serves as an intermediary that absorbs data manipulation operations during the merge process. Instead of requiring locks on the first delta part, write operations are redirected to the shadow part, eliminating lock management complexity while maintaining full data manipulation capability. The shadow part acts as a dedicated workspace for modifications during merges.
Solution Approach 2:
The delta part functionality is segmented into the first delta part (for reading and historical data) and the shadow second delta part (for active manipulation during merges). This segmentation allows data manipulation to occur in the shadow part without conflicting with read operations on the first delta part, reducing lock complexity while maintaining operational capability.
Data Source
AI summary
Improved availability of data in columnar tables can be provided via use of additional data structures, such as for example a second delta part, a shadow second delta part, and a second main part, that are employed during a merge operation of a first delta part into a main part. Related apparatus, systems, techniques and articles are also described.