Database Management Using Append-Only Storage Logs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional 'update-in-place' processes are inefficient and shorten the lifespan of append-only storage devices like SSDs, as they require frequent random writes, which is not well-suited for modern data centers adopting these devices.
Innovation Solution
Implementing a system that interfaces between a database management layer and append-only storage devices, using a storage abstraction layer to manage page writes and reads by appending component blocks to an append-only storage log, allowing for sequential writing and efficient data retrieval, and incorporating a log cleaner to consolidate and clear obsolete data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If update-in-place process is used to update records in database, then write speed is improved for legacy storage devices, but lifespan of append-only storage devices deteriorates due to frequent random writes
Solution Approach 1:
The patent segments the storage system into a database management layer and a storage layer with append-only storage devices. By separating the database operations from the physical storage operations and using an append-only log structure, the system avoids random writes to the storage device while maintaining database functionality. This segmentation resolves the contradiction by allowing the database layer to perform logical updates while the storage layer only performs sequential appends, thereby improving write speed without sacrificing device lifespan.
Solution Approach 2:
The patent introduces an intermediary log structure between the database management system and the append-only storage device. This log acts as a mediator that translates database update operations into sequential append operations. The intermediary log buffer and log writer component convert random write requests from the database into sequential writes to the storage device, resolving the contradiction by decoupling the database update operations from direct storage writes and enabling both high write speed and extended device lifespan.
2Productivity
If append-only storage devices are used in modern data centers, then power consumption is reduced and read/write speed is improved, but compatibility with legacy database systems deteriorates
Solution Approach 1:
The patent introduces a storage abstraction layer as an intermediary between legacy database systems and append-only storage devices. This abstraction layer includes components such as the log writer, log reader, and log cleaner that translate legacy database operations into append-only storage operations. The intermediary layer maintains compatibility with legacy database interfaces while enabling efficient utilization of append-only storage devices, thereby resolving the contradiction by allowing legacy systems to access modern storage technology without requiring system-wide changes.
Solution Approach 2:
The patent creates a universal interface layer that can work with both legacy database systems and modern append-only storage devices. The storage abstraction layer provides multi-functional capabilities, supporting various database operations (reads, writes, updates) while adapting to the append-only nature of the storage device. This universality allows the system to maintain compatibility with legacy databases while fully utilizing the performance benefits of append-only storage, resolving the adaptability contradiction.
3Device complexity
If conventional database systems are used with append-only storage devices, then implementation simplicity is maintained, but data retrieval efficiency deteriorates
Solution Approach 1:
The patent implements preliminary action through the log cleaner component that proactively consolidates and repositions log data before it is needed for retrieval. The log cleaner performs background consolidation operations, moving frequently accessed log records to optimal positions and removing obsolete entries. This preliminary action improves data retrieval efficiency by ensuring that hot data is readily accessible while maintaining the simplicity of the append-only storage interface, resolving the contradiction between implementation simplicity and retrieval efficiency.
Solution Approach 2:
The patent implements periodic action through scheduled log consolidation and cleaning operations. The log cleaner performs periodic maintenance tasks to optimize the log structure, consolidating fragmented data and removing obsolete records at regular intervals. This periodic optimization maintains high data retrieval efficiency without requiring continuous complex management operations, thereby resolving the contradiction by achieving efficient retrieval through periodic rather than continuous intervention.
Data Source
AI summary
An apparatus is disclosed. The apparatus may be implemented in a database node or a storage node and includes one or more processors and memory storing instructions for causing the processor to perform a number of operations. Responsive to a page write request, the apparatus determines an identifier corresponding to the requested page, sends component blocks corresponding to the page to a storage node for appending to an append-only storage log of the storage node, receives the physical location of the stored component blocks, and associates the physical storage location of the stored component blocks with the logical identifier corresponding to the page.