Off-node database compaction via snapshot isolation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database compaction processes consume compute resources, leading to increased latency and reduced availability for serving requests, as they require the database node to dedicate resources for file compaction.
Innovation Solution
Implementing a separate compaction processing node within a compute infrastructure that generates a snapshot of the database node's data store, allowing for off-node compaction without using the database node's resources, thereby reducing compute resource usage and improving latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If compaction is performed on the database node itself, then the compaction process can directly access and process datafiles, but compute resources are consumed resulting in increased latency and reduced availability for serving requests
Solution Approach 1:
The compaction processing function is extracted from the database node and performed on a separate compaction processing node. The database node generates a snapshot of its data store and transfers it to the compaction processing node, which then performs the compaction operation independently. This separation allows the database node to serve requests without being blocked by compaction operations, thereby reducing request serving latency while maintaining compaction productivity.
Solution Approach 2:
A snapshot mechanism serves as an intermediary between the database node and the compaction processing node. The database node creates a snapshot of its data store and transfers this snapshot to the compaction processing node for processing. This intermediary approach allows the compaction operation to proceed on separate hardware resources while the original database node remains available to serve requests, resolving the contradiction between compaction efficiency and request serving latency.
2Device complexity
If multiple SSTables are compacted into a single compacted SSTable, then the number of SSTables consulted for access is reduced and storage space is reclaimed, but compute resources are consumed resulting in fewer resources available to operate the database
Solution Approach 1:
The computationally intensive compaction operation is extracted from the database node and executed on a separate compaction processing node. The database node only needs to generate snapshots and transfer data, while the actual merging of multiple SSTables into compacted SSTables is performed by the separate compaction processing node. This reduces the compute resource usage of the database node while still achieving the benefits of reduced SSTable count and reclaimed storage space.
Solution Approach 2:
The compaction system is segmented into two distinct components: the database node that manages data storage and retrieval, and the compaction processing node that handles file consolidation. By segmenting the compaction function into a separate component that operates on snapshots transferred from the database node, the system reduces compute resource contention while maintaining the ability to compact multiple SSTables into fewer files.
3Reliability
If a snapshot of the data store is created and transferred to a separate data store for compaction, then the database node's compute resources are preserved for serving requests, but additional storage infrastructure and data transfer operations are required
Solution Approach 1:
A snapshot mechanism acts as an intermediary that enables the database node to preserve its compute resources for serving requests while still performing compaction. The snapshot is created locally and transferred to a separate data store that is attached to the compaction processing node. This intermediary approach maintains request serving availability while the additional storage infrastructure is only activated when compaction operations are needed, rather than being permanently required.
Solution Approach 2:
The database node creates a snapshot of its data store before transferring data to the compaction processing node. This preliminary action allows the compaction operation to proceed on a copy of the data without affecting the original data store or the database node's ability to serve requests. The snapshot creation is a preparatory step that enables the separation of compiction processing from the database node while maintaining system availability.
Data Source
AI summary
Embodiments relate to a compacting datafiles generated by a database node using a compaction processing node with separate compute resources. The database node generates datafiles and stores the datafiles in a data store. To perform compacting of the datafiles, a snapshot of the data store is created and stored in a snapshot store separate from the data store. The compaction processing node is initiated and attached with the snapshot store. The compaction processing node generates a compacted datafile that is stored in the snapshot store. The database node replaces the data store with the snapshot store, and writes additional datafiles using the snapshot store as a new data store. The compaction processing node may be an instance of a cloud compute infrastructure that is initiated to perform the compaction to reduce compute resource usage by the database node.