Buffer Cache on Non-Volatile Memory for Database Durability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database systems face challenges in utilizing non-volatile memory effectively due to differences in behavior between volatile DRAM and persistent storage, leading to issues with data durability and ACID properties when transitioning from DRAM to NVRAM, resulting in unpredictable database states during crashes.
Innovation Solution
Implementing a buffer cache with DRAM-based buffer headers that manage access to data stored in NVRAM by using direct and DRAM-copy mappings, allowing processes to access data directly in NVRAM or through copies in DRAM, while maintaining persistence and controlling access permissions to ensure data integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the database is stored completely in NVRAM, then access speed is improved compared to disk storage, but data durability and ACID properties cannot be guaranteed when processes crash before changes are complete
Solution Approach 1:
The patent introduces buffer headers stored in DRAM as an intermediary layer between processes and the NVRAM-based file system. These buffer headers manage access permissions and track the state of data blocks, enabling the system to coordinate between the speed of NVRAM and the reliability requirements of persistent storage. The buffer headers act as a control mechanism that ensures ACID properties are maintained while allowing fast NVRAM access.
Solution Approach 2:
The patent changes the persistence parameter of the buffer cache from volatile (traditional DRAM buffer cache) to persistent (NVRAM-based), fundamentally altering how the system balances speed and reliability. By making the buffer cache persistent, the system can tolerate crashes without losing buffered data, thereby maintaining ACID properties while enjoying faster NVRAM access speeds.
2Device complexity
If DRAM-enabled database systems directly replace DRAM and persistent storage with NVRAM, then device complexity is reduced, but the system fails to maintain ACID properties due to unpredictable database states after crashes
Solution Approach 1:
The patent segments the storage architecture into distinct layers: a persistent file system layer in NVRAM that handles durability, and a buffer cache layer with buffer headers in DRAM that handles access control and transaction management. This segmentation allows each layer to specialize in its function, maintaining ACID properties through proper coordination while simplifying the overall architecture compared to traditional multi-layer storage systems.
Solution Approach 2:
The buffer headers serve multiple functions: they act as access control mechanisms, track the persistence state of data blocks, manage transaction boundaries, and coordinate between the file system and database processes. This multi-functionality reduces the need for separate mechanisms for each concern, thereby reducing overall system complexity while maintaining reliability.
3Productivity
If a buffer cache is implemented in NVRAM with direct process access, then I/O overhead is minimized, but data integrity cannot be ensured without access control mechanisms
Solution Approach 1:
The buffer headers enable the file system to self-manage access control and data integrity without requiring additional intermediary layers. The buffer headers track which processes have access to which data blocks and enforce persistence rules, allowing the system to maintain data integrity while enabling direct I/O operations. This self-service approach minimizes I/O overhead while ensuring reliability.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Techniques for implementing a buffer cache for a persistent file system in non-volatile memory is provided. A set of data is maintained in one or more extents in non-volatile random-access memory (NVRAM) of a computing device. At least one buffer header is allocated in dynamic random-access memory (DRAM) of the computing device. In response to a read request by a first process executing on the computing device to access one or more first data blocks in a first extent of the one or more extents, the first process is granted direct read access of the first extent in NVRAM. A reference to the first extent in NVRAM is stored in a first buffer header. The first buffer header is associated with the first process. The first process uses the first buffer header to directly access the one or more first data blocks in NVRAM.