Erasure-Coded Data Storage With Flexible Redundancy and Fast Rebuilds
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage methods for video surveillance, such as RAID and backup schemes, face inefficiencies in disk space usage, redundancy, and flexibility, particularly when handling data loss or corruption, and require significant additional capacity or fixed disk configurations.
Innovation Solution
A computer-implemented method that splits data into N pieces and generates M redundancy pieces, storing them on separate storage media at the same offset within a file set, allowing for flexible redundancy levels and efficient use of disk space, with the option to vary the number of disks and redundancy based on data importance and retention schedules.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full duplicates of the data are stored (RAID1), then data reliability is improved, but disk capacity requirement doubles
Solution Approach 1:
The data is segmented into N pieces and distributed across multiple storage media, with only M redundancy pieces stored separately. This segmentation allows the system to achieve reliable data recovery without storing complete duplicates, reducing the total storage capacity requirement compared to RAID1 mirroring.
Solution Approach 2:
The system changes the redundancy parameter from full duplication (RAID1) to erasure coding with configurable N and M values. By adjusting these parameters, the system can optimize the balance between data reliability and storage capacity utilization, achieving reliable data protection with lower storage overhead.
2Quantity of substance
If RAID 5 or 6 is used, then storage overhead is reduced, but flexibility in varying redundancy levels is lost
Solution Approach 1:
The system implements dynamic redundancy configuration where N and M can be adjusted based on data importance and retention schedules. This dynamic approach allows the system to adapt redundancy levels for different data sets, providing flexibility that fixed RAID configurations cannot offer while maintaining efficient storage overhead through erasure coding.
3Device complexity
If fixed size disk sets are used, then system complexity is reduced, but ability to handle drive failures with varied disk availability is limited
Solution Approach 1:
The erasure coding system with separate file sets and offsets provides a universal storage framework that can handle various disk availability scenarios. The same basic structure supports different N and M configurations, allowing the system to adapt to different numbers of available disks and failure scenarios without requiring complex specialized configurations for each case.
4Reliability
If data is rebuilt after drive failure, then data reliability is restored, but performance bottlenecks occur
Solution Approach 1:
By segmenting data into pieces stored at the same offset in separate files across multiple storage media, the system enables parallel reconstruction. When a drive fails, the redundant pieces from other drives can be read simultaneously and used to reconstruct the missing data, improving rebuild performance compared to sequential reconstruction methods.
Data Source
AI summary
A computer-implemented method of storing an item of data across a plurality of storage media, the method comprising the steps of: receiving an item of data to be stored; splitting the item of data into N pieces of data; generating M redundancy pieces of data, usable to rebuild the item of data; storing each of the N pieces of data and M redundancy pieces of data on separate storage media of the plurality of storage media at a same offset within a file of the respective storage medium, the files of the storage media containing the N pieces of data and the M redundancy pieces of data being associated as a file set; and storing, separately to the N pieces of data and M redundancy pieces of data, the offset and the file set.