Hybrid Shadow Paging with Buffered Patches for Low-Latency Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems struggle to efficiently manage long transactions with time-travel capabilities while minimizing computing resources and I/O overhead, particularly in creative authoring applications where multiple users access and modify data concurrently, requiring improved strategies for atomicity, consistency, isolation, and durability.
Innovation Solution
A hybrid shadow paging and logging method that buffers modified pages and creates patches based on thresholds, using a mapping table and append-only event log storage to reduce I/O costs and latency, ensuring atomicity, isolation, and durability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If shadow paging is used to support long transactions with time-travel capabilities, then atomicity and consistency are improved, but I/O resources and latency are worsened due to waiting for all pages to be written to disk
Solution Approach 1:
The patent segments the shadow paging process by introducing a buffer layer between memory and disk. Modified pages are first written to a buffer in memory, and only flushed to disk when the buffer reaches a threshold. This segmentation allows transactions to commit quickly to the buffer while disk I/O is deferred, resolving the contradiction between atomicity (ensured by buffer flushing) and latency (reduced by deferred I/O).
Solution Approach 2:
The patent performs preliminary actions by pre-allocating buffer space and pre-processing page modifications in memory before disk commitment. The buffer acts as a preliminary staging area where pages are prepared for disk writing, allowing transactions to proceed without waiting for actual disk flushes, thus reducing latency while maintaining atomicity through subsequent buffer flushing.
2Reliability
If shadow paging maintains two versions of files (shadow and current), then consistency and durability are improved, but device complexity and storage overhead are worsened
Solution Approach 1:
The patent applies local quality by maintaining full shadow copies only for specific pages that have been modified, rather than maintaining complete shadow versions of all files. The buffer tracks which pages need shadowing, and only those pages are flushed to disk as shadow pages. This selective approach maintains consistency for modified pages while reducing overall storage overhead and system complexity.
Solution Approach 2:
The patent implements discarding and recovering by allowing shadow pages to be discarded from the buffer once they have been successfully flushed to disk. The buffer dynamically manages shadow page retention, discarding them after successful writing to reclaim space, thus reducing storage overhead while maintaining consistency through the shadow paging mechanism.
3Ease of operation
If all database states are made accessible with equal cost for time-travel queries, then ease of operation is improved, but productivity is worsened due to increased I/O operations
Solution Approach 1:
The patent implements periodic action by flushing the buffer to disk at regular intervals when a threshold is reached, rather than after every transaction. This periodic flushing creates discrete checkpoints in the database state history, allowing time-travel queries to access these checkpoints efficiently without requiring I/O operations for every intermediate state, thus balancing access efficiency with reduced I/O overhead.
Solution Approach 2:
The patent uses copying by creating shadow copies of modified pages in the buffer and flushing them to disk as discrete state snapshots. These copied shadow pages represent specific database states that can be efficiently accessed for time-travel queries, providing equal access cost for historical states while minimizing I/O through selective copying rather than continuous replication.
Data Source
AI summary
A computer-implemented method for storing a database state. The method comprises providing a database, receiving by the database one or more write events, logging each write event, each logged write event thus forming a new state on the database, buffering pages modified or created by the write events, and creating a patch by flushing to a database storage the buffered pages if a threshold has been met.