NoSQL Deletion Compactor Reduces CPU and Disk Writes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional compaction processes for large NoSQL databases with static data are inefficient, leading to excessive resource usage and discouraging frequent compaction due to iterative copying of data, high CPU usage, and excessive disk writes.
Innovation Solution
A method and system for efficiently compacting a NoSQL database by utilizing a deletion compactor process that identifies and groups files with deleted records, compacting only after a threshold level of deletions occurs, without relying on indexing, and performing compaction during low-throughput times to minimize resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional compaction processes are used for large NoSQL databases with static data, then compaction can be performed, but excessive CPU usage, excessive disk writes, and excessive temporary disk usage occur
Solution Approach 1:
The patent extracts and separates the identification of deleted records from the compaction process itself. By using a tombstone index to pre-identify deleted records before compaction, the system removes unnecessary copying of unchanged data, thereby reducing CPU usage and improving compaction efficiency for databases with static data.
2Productivity
If conventional compaction processes are used for large NoSQL databases with static data, then compaction can be performed, but excessive CPU usage, excessive disk writes, and excessive temporary disk usage occur
Solution Approach 1:
The patent extracts the identification of deleted records using a tombstone index before the compaction process. This allows the system to skip copying unchanged data blocks, significantly reducing the number of disk writes required during compaction for databases with static data.
3Quantity of substance
If compaction is performed frequently to reclaim space, then disk space reclamation is improved, but resource usage becomes excessive
Solution Approach 1:
The patent performs preliminary identification of deleted records using a tombstone index before compaction. This preliminary action allows the system to efficiently determine which data blocks need to be compacted, enabling more frequent compaction operations with reduced resource usage by avoiding unnecessary copying of unchanged data.
4Quantity of substance
If iterative copying of data is performed during compaction, then obsolete data is removed, but excessive temporary disk usage occurs
Solution Approach 1:
The patent extracts and processes only the portions of data that contain deleted records by using the tombstone index to identify specific data blocks. This selective approach removes obsolete data while minimizing temporary disk usage by avoiding the creation of complete data copies during compaction.
Data Source
AI summary
System and method to compact a NoSQL database, the method including: receiving, by a receiver coupled to a processor, an indication of a record to delete in the NoSQL database; for each file in the NoSQL database, perform the steps of: if said file does not contain the record to delete, placing said file in a first memory; if said file contains the record to delete: placing said file in a second memory; searching whether the record to delete from said file in the second memory matches a record in one or more files in the first memory; and if a searched files in the first memory contain the record to delete from said file in the second memory, compacting said file in the second memory with the files in the first memory that contain the record to delete.