Distributed Storage Garbage Collection for Versioned Rowblocks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed database systems face challenges in managing uneven workloads and transaction conflicts due to the lack of record-level processing and efficient data management, leading to increased computational burden and reduced performance.
Innovation Solution
Implementing a record-aware distributed storage system that uses rowblocks for data management, supporting versioned records, transaction conflict detection, and multiversion concurrency control, allowing the system to handle transactions efficiently and reduce the computational load on database access applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If distributed database systems are implemented to scale processing capacity, then service availability and processing capacity are improved, but uneven workload distribution and transaction conflicts increase
Solution Approach 1:
The patent segments the database system into separate compute nodes and storage nodes, with further segmentation of storage into rowblocks and record versions. This segmentation allows independent scaling and management of compute and storage resources, addressing the uneven workload distribution problem by enabling granular control over resource allocation and workload balancing across the distributed system.
2Reliability
If distributed storage systems are used to increase service availability, then system reliability is improved, but computational burden and performance overhead increase
Solution Approach 1:
The storage nodes autonomously perform garbage collection by detecting and reclaiming deleted record versions without requiring coordination from compute nodes. This self-service mechanism reduces the computational burden on the distributed system by eliminating the need for compute nodes to participate in garbage collection operations, thereby reducing overall computational overhead while maintaining data reliability through distributed storage.
3Productivity
If record-level processing is implemented to manage transactions efficiently, then transaction handling performance is improved, but system complexity and operational overhead increase
Solution Approach 1:
The patent introduces rowblocks as an intermediary data structure between compute nodes and individual records. Rowblocks group multiple record versions together and manage them as a unit, simplifying the operations needed for transaction processing. This intermediary layer enables efficient record-level processing by allowing compute nodes to work with rowblocks rather than individual records, reducing the complexity of transaction management while maintaining high performance.
Data Source
AI summary
Scalable garbage collections is performed by a distributed storage system. Garbage collection events are detected by a distributed storage system for different portions of a table. Garbage collection is performed for individual ones of the different portions of the table responsive to detecting the garbage collection events, including identifying one or more versions of a record to reclaim from the different portions of the table based on transaction status information and reclaiming the one or more versions of the record.