Double Buffering for Atomic Real-Time Data Persistence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing medical systems face challenges in persistently storing real-time electrophysiological data during medical procedures, particularly in ensuring data integrity and continuous processing without delays, especially in the event of system failures or power outages.
Innovation Solution
Implementing double buffering with atomic transactions, where two buffers and a variable are configured in memory and storage, allowing for seamless swapping and committing of data streams to a persistent storage medium, ensuring data integrity and continuous processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If real-time electrophysiological data is continuously streamed to persistent storage, then data persistence is improved, but system reliability deteriorates due to potential data loss during power outages or system failures
Solution Approach 1:
The persistent storage is segmented into multiple chunks, where each chunk represents a discrete unit of data that can be independently managed and committed. This segmentation allows the system to commit complete chunks atomically, ensuring that if a failure occurs, only incomplete chunks are affected while other chunks remain intact and recoverable.
Solution Approach 2:
The system performs preliminary actions by buffering data in memory before committing to persistent storage. The buffer acts as a preliminary holding area where data can be staged and prepared for atomic chunk commits. This preliminary buffering allows the system to maintain data in a recoverable state temporarily, reducing the risk of data loss during failures.
2Reliability
If data is committed frequently to persistent storage, then data integrity is improved, but processing speed deteriorates due to I/O operations
Solution Approach 1:
The system accumulates data in a buffer in memory as a preliminary step before committing to persistent storage. This preliminary buffering allows multiple data points to be collected and prepared for commitment without immediate I/O operations, thereby maintaining processing speed while ensuring data integrity through periodic atomic commits.
Solution Approach 2:
Instead of committing every single data point immediately, the system uses partial action by committing chunks of data that meet certain criteria (e.g., size threshold or time interval). This partial commitment approach balances data integrity with processing efficiency, avoiding the overhead of frequent I/O operations while still maintaining reliable persistence.
3Productivity
If a single buffer is used for data storage, then device complexity is reduced, but productivity deteriorates due to processing delays
Solution Approach 1:
The buffer is segmented into multiple chunks, each representing a discrete unit of data storage. This segmentation allows the system to manage multiple buffer regions independently, enabling continuous processing by switching between different buffer chunks while maintaining organized and manageable buffer structures.
Solution Approach 2:
The buffer management is made dynamic through the use of a circular buffer approach where buffer chunks are continuously allocated, filled, committed, and reused. This dynamic buffer management enables continuous data processing without waiting for individual buffer commits, thereby improving productivity while maintaining manageable complexity through systematic buffer rotation.
4Reliability
If atomic transactions are used for chunk commits, then data integrity is improved, but device complexity increases due to transaction management
Solution Approach 1:
The data storage is segmented into discrete chunks that can be committed as atomic units. This segmentation simplifies transaction management by providing clear boundaries for atomic commits, making it easier to implement and manage transactions compared to handling continuous data streams. Each chunk becomes a self-contained unit that can be committed independently with clear success or failure states.
Data Source
Figure 1
Figure 2
Figure 3A~3B
AI summary
A method, including configuring, in a memory, a first buffer to store first buffer data, a second buffer to store second buffer data, and a variable indicative of a persisted size of real-time data persisted to a storage device. On the storage device, a file is configured to store the real-time data and a field is configured to store the variable. A stream of the real-time data is received, and the stream is appended to the first buffer data. Upon meeting a write criteria, the first buffer data in is swapped with the second buffer data, a buffer size of the second buffer data is added to the variable, and the second buffer data is conveyed from the second buffer to a write cache. Upon meeting a commit criteria, the stream stored in the write cache is appended to the file, and the variable is persisted to the field.