Compaction-Less Merkle Trie Storage to Reduce Write Amplification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems face inefficiencies due to redundant node writes, particularly in Merkle trie data storage, leading to increased disk I/O and storage overhead, especially in systems using write-ahead logs and log-structured merge backend databases.
Innovation Solution
A compaction-less database structure that stores Merkle trie nodes directly on-disk, allowing nodes to be written once and overwritten if changed or marked for deletion, eliminating the need for periodic compaction and reducing redundant writes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If nodes are written multiple times in current database systems, then data integrity is maintained, but disk I/O increases and storage efficiency deteriorates
Solution Approach 1:
The system performs preliminary actions by maintaining a future-delete list that tracks nodes scheduled for deletion before they are actually removed. This allows the database to proactively identify and mark redundant nodes for deletion, preventing unnecessary writes and reducing disk I/O operations while maintaining data integrity through controlled node lifecycle management.
Solution Approach 2:
The system discards redundant node copies by implementing a future-delete list mechanism that identifies and removes duplicate or expired nodes from the database. This principle recovers storage space by eliminating unnecessary node replicas that would otherwise consume disk I/O resources, while maintaining the required data integrity through selective deletion of only those nodes that are no longer needed.
2Quantity of substance
If periodic compaction is performed to reduce storage overhead, then storage efficiency improves, but system complexity and operation difficulty increase
Solution Approach 1:
The system implements self-service by automatically managing node deletion through a future-delete list that is maintained and updated as part of normal database operations. This eliminates the need for separate, complex periodic compaction operations, as the system continuously and autonomously identifies and removes redundant nodes, reducing storage overhead without increasing operational complexity.
Solution Approach 2:
By performing preliminary identification and marking of nodes for deletion through the future-delete list, the system prepares the database structure in advance, making subsequent storage optimization automatic and simple. This preliminary action transforms the complex periodic compaction task into a continuous, automated process that reduces storage overhead without requiring complex operational intervention.
3Reliability
If write-ahead logs and log-structured merge backends are used to ensure data consistency, then reliability improves, but write amplification increases
Solution Approach 1:
The system applies discarding and recovering by implementing a future-delete list that identifies and removes redundant node writes before they occur. This mechanism discards unnecessary write operations that would otherwise amplify disk I/O, while recovering write efficiency by ensuring each node is written only when necessary. This maintains data consistency through controlled node lifecycle management without the write amplification associated with traditional write-ahead logs and log-structured merge backends.
Data Source
AI summary
Various aspects of the subject technology relate to systems, methods, and machine-readable media for implementing a compaction-less database structure. Various aspects may include identifying nodes in a most recent revision of a database structure stored on disk corresponding to locations of node changes in a current proposal. Aspects may also include persisting node addresses for nodes that have changed from the most recent revision in the current proposal to a future-delete list (also kept on disk). Aspects may also include writing only the new nodes including a root node to disk. Aspects may also include updating a root address on disk based on the root node. Aspects may also include committing the proposal as a new revision of the database structure.