Distributed Storage with Rateless Erasure Coding for Redundancy Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage solutions, such as RAID and replication-based systems, are inefficient in using storage capacity for redundancy and lack scalability to meet growing data needs, while conventional distributed storage systems face challenges in ensuring data availability and flexibility.
Innovation Solution
The method employs rateless erasure encoding using online codes to fragment data into blocks, encode them, and distribute them across multiple storage entities, allowing for efficient and scalable redundancy with flexible distribution policies, ensuring data recovery even if some entities fail or degrade.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If RAID or replication-based systems are used to provide redundancy, then data reliability is improved, but storage capacity efficiency deteriorates
Solution Approach 1:
The data set is divided into multiple data blocks by a fragmentation entity, which are then encoded into multiple encoded blocks. These encoded blocks are distributed across multiple storage entities, allowing the system to achieve both redundancy and efficient storage capacity utilization through segmented data distribution.
Solution Approach 2:
The system employs rateless erasure encoding algorithms that can dynamically adjust the number of encoded blocks generated from the original data blocks. This parameter change capability allows the system to optimize the balance between redundancy level and storage capacity efficiency based on specific requirements.
2Quantity of substance
If conventional distributed storage systems are used to increase storage capacity, then scalability is improved, but data availability and flexibility deteriorate
Solution Approach 1:
The distributed storage system is designed to perform multiple functions: it can store data across multiple storage entities, provide redundancy through encoding, enable flexible data retrieval, and adapt to varying storage capacity requirements. The rateless erasure encoding allows the same system to serve different scalability needs while maintaining data availability.
Solution Approach 2:
The system dynamically adjusts the encoding process to generate the appropriate number of encoded blocks based on the desired redundancy level and storage capacity. The distribution entity can flexibly allocate encoded blocks across storage entities, and the system can adapt to changes in storage availability without requiring fixed redundancy configurations.
3Quantity of substance
If data is distributed across multiple storage entities to reduce required capacity, then storage efficiency is improved, but system complexity increases
Solution Approach 1:
The system employs self-service mechanisms where the encoding and distribution processes are automated. The encoding entity automatically generates the required number of encoded blocks, and the distribution entity automatically distributes them across storage entities according to the desired redundancy level, reducing the need for manual configuration and management.
Data Source
AI summary
The present invention is directed to a method of storing a data set in a storage system. Said storage system comprises a plurality of storage entities. The method comprises the following steps. A step of forming at least one data set block from said data set using a fragmentation entity. A step of encoding said at least one data set block by means of an encoding entity, wherein said encoding entity provides a plurality of encoded blocks based on said at least one data set block at an output of said encoding entity. And a step of distributing said encoded blocks amongst at least two of said storage entities using a distribution entity for storage thereof by said storage entities. Said encoding entity uses a rateless erasure encoding algorithm based on rateless codes for encoding said at least one data set block. The invention is further directed to a storage system and a computer program product.