Distributed Database Compaction via Asynchronous Thread Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud-based data warehouses and database systems face challenges in efficiently processing concurrent transactions and compacting transactional data in a scalable and performant manner without affecting data accuracy or integrity.
Innovation Solution
The implementation of a network-based database system that employs a two-level transaction hierarchy, multi-version concurrency control, and targeted isolation levels to manage concurrent transactions, combined with a separate compactor that tracks oldest transactions' read timestamps and periodically sweeps the database to compact dead versions of objects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If transactional processing is implemented in cloud-based data warehouses, then operational capabilities are enhanced, but data compaction becomes difficult and system complexity increases
Solution Approach 1:
The patent divides the database system into distinct components: transaction processing subsystem and compaction subsystem. Each subsystem operates independently with dedicated threads and protocols, allowing complex transactional processing without compromising the simplicity and efficiency of the compaction process. The segmentation enables both functionalities to coexist without interfering with each other's performance.
Solution Approach 2:
The patent introduces version identifiers and transaction logs as intermediary mechanisms that mediate between transactional operations and data storage. These intermediaries track all changes without requiring the compaction process to understand transactional semantics, thereby reducing system complexity while maintaining full transactional capability.
2Adaptability or versatility
If concurrent transactions are processed, then operational versatility improves, but data integrity and accuracy become harder to maintain
Solution Approach 1:
The patent implements preliminary version tagging and transaction logging before actual data modifications occur. Each transaction is assigned a unique version identifier and logged with its intended changes. This preliminary action ensures that even during concurrent operations, the system can track and verify data integrity by comparing version histories, maintaining reliability while enabling operational versatility.
Solution Approach 2:
The compaction subsystem continuously monitors transaction logs and version identifiers to detect conflicts and ensure data integrity. When concurrent transactions create conflicting versions, the feedback mechanism identifies these conflicts and resolves them through coordinated compaction operations, thereby maintaining data accuracy while allowing high levels of concurrent operational versatility.
3Quantity of substance
If data compaction is performed frequently, then storage efficiency improves, but query performance and data accuracy may be affected
Solution Approach 1:
The patent implements periodic compaction operations that run at predetermined intervals rather than continuously. During these periodic compaction cycles, the system consolidates data versions and removes obsolete records to improve storage efficiency. The periodic nature ensures that compaction does not interfere with ongoing queries, as compaction operations are scheduled during low-activity periods or run in parallel with query processing.
Solution Approach 2:
The compaction frequency and intensity are dynamically adjusted based on system workload and storage conditions. When query activity is high, compaction operations are reduced or paused to maintain query performance. When storage space becomes constrained or during low-activity periods, compaction is intensified to improve storage efficiency. This dynamic adaptation allows the system to optimize both storage efficiency and query performance based on real-time conditions.
Data Source
AI summary
The subject technology performs, using asynchronous threads of execution nodes of a distributed database, asynchronous transformation and compaction of key-value pairs of data that correspond to a plurality of transactional queries, the asynchronous threads transforming the key-value pairs from an initial format to a commit format that includes a commit time of data committed to the distributed database, the asynchronous threads compacting the key-value pairs by deleting the key-value pairs in the initial format. The subject technology processes, using transactional threads in the execution nodes, the plurality of transactional queries to generate results data, the asynchronous threads performing the asynchronous transformation and compaction while the transactional threads generate the results data. The subject technology stores the results data.