Circular Stream File Layout for Fragmentation-Free Data Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems like XFS and ext4 face performance issues due to disk fragmentation and scattered file data, particularly when handling continuous data streams such as surveillance recordings, which require real-time storage without frequent modifications.
Innovation Solution
A data storage system that allocates defined memory blocks for data storage, stores data in a circular manner, ensuring data blocks are filled sequentially and overwritten in a loop, maintaining data chronology and reducing disk seeks, and implements a First-In-First-Out (FIFO) mechanism for automatic deletion of older data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If general-purpose file systems (XFS, ext4) are used to accommodate various file operations, then versatility is improved, but disk fragmentation and scattered file data occur leading to performance degradation
Solution Approach 1:
The patent segments the storage system into two distinct components: a general-purpose file system layer that handles diverse file operations, and a specialized stream file system layer that optimizes continuous data storage. The stream file system divides data into fixed-size blocks and stores them sequentially in a circular buffer structure, isolating the performance-critical path from fragmentation issues while maintaining versatility through the file system interface.
Solution Approach 2:
The stream file system acts as an intermediary layer between the general-purpose file system and the physical storage medium. It receives continuous data streams, processes them through its optimized circular buffer mechanism, and presents them as regular files to the upper layer, thereby shielding the storage performance from the overhead of general file system operations.
2Adaptability or versatility
If data is stored in conventional file systems with frequent modifications, then adaptability is improved, but disk seek overhead increases reducing storage speed
Solution Approach 1:
The stream file system implements continuous writing to a circular buffer structure where data blocks are sequentially written in a loop. This eliminates random disk seeks by maintaining continuous sequential access patterns, as the write head simply continues to the next block in the circular arrangement. The system sustains this continuous useful action indefinitely, making it ideal for surveillance and logging applications.
Solution Approach 2:
The circular buffer structure implements periodic action by cycling through fixed-size data blocks in a predetermined sequence. When the end of the buffer is reached, the system seamlessly wraps around to the beginning, creating a periodic write pattern that eliminates the need for random disk positioning. This periodic structure ensures consistent storage performance regardless of data volume.
3Quantity of substance
If older data is retained in storage, then data completeness is improved, but storage space is consumed reducing efficiency for new data
Solution Approach 1:
The circular buffer mechanism automatically discards older data blocks when the buffer becomes full, overwriting them with new data. This eliminates the need for manual data management while maintaining storage efficiency. The system recovers storage space seamlessly by reusing allocated blocks in a continuous cycle, ensuring that storage capacity is always fully utilized without fragmentation or wasted space.
Data Source
AI summary
A method is disclosed. In the method, a set of data blocks can be stored. A continuous stream of data can be received. First data from the continuous stream of data can be stored in a first subset of data blocks. Second data from the continuous stream of data can be stored in a second subset of data blocks. Responsive to determining each of the set of data blocks is filled with data, data in a third subset of data blocks can be overwritten with data from the continuous stream of data.