Clustered File Persistence for Torn Write Prevention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In computing devices, persisting small datasets at high frequency leads to performance and durability challenges, particularly due to the risk of 'torn writes' when datasets are collocated in the same file and changed at independent rates, resulting in incomplete data portions within a file.
Innovation Solution
A mechanism is implemented to maintain datasets in a single file, allowing for sequential I/O operations, with each file divided into clusters of a designated size, ensuring durability by preventing co-location of changes within the same cluster, and using non-cached, write-through I/O operations to avoid data corruption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If datasets are stored in different files to prevent torn writes, then data durability is improved, but I/O performance deteriorates due to increased number of I/O requests
Solution Approach 1:
The patent divides a single file into multiple clusters, where each cluster can independently store a dataset. This segmentation allows multiple datasets to coexist in one file while maintaining isolation through cluster boundaries, thus preventing torn writes without requiring separate files for each dataset.
Solution Approach 2:
The patent introduces the cluster dimension within a file system, transforming the traditional file-level storage model into a hierarchical model with file and cluster levels. This allows datasets to be organized in clusters within files, enabling both durability (through cluster isolation) and performance (through reduced I/O operations compared to separate files).
2Productivity
If multiple datasets are collocated in the same file, then I/O performance is improved by reducing number of I/O requests, but data durability deteriorates due to risk of torn writes
Solution Approach 1:
By segmenting the file into distinct clusters and assigning each dataset to its own cluster, the patent enables multiple datasets to be collocated in the same file while maintaining durability. The cluster boundaries act as isolation barriers that prevent torn writes even when datasets are updated independently.
Solution Approach 2:
The patent applies local quality by giving each cluster specific properties (such as cluster flags and metadata) that distinguish it from other clusters. This allows the filesystem to track and manage each dataset's location and state independently within the file, ensuring durability while enabling efficient I/O operations.
3Ease of operation
If per-file clustering is used to manage objects, then object management is simplified, but all objects must be stored from the beginning causing significant impact on database generation time
Solution Approach 1:
The patent pre-allocates clusters within files and prepares the file structure in advance, but does not require all data to be written immediately. The filesystem can manage the pre-configured cluster structure efficiently while allowing incremental data population, thus avoiding the time penalty of generating entire databases from scratch.
Data Source
Figure 1
Figure 2
Figure 3A~3C
AI summary
The systems and methods disclosed herein relate to using the clusters of a file to store versioning of a dataset. When the dataset is initially stored, a file is created that is twice the size of the dataset. The file may include one cluster (or a first set of clusters) that is marked as active and a second cluster (or a second set of clusters) that are marked inactive. The dataset is initially saved to the active cluster(s), and a version number is stored with the dataset. When the dataset is next saved, an application scans the file to determine whether there is (or are) an inactive cluster(s). If there is an inactive cluster(s) the second version of the dataset is saved to the inactive clusters. Both clusters are then marked active.