Storage Cluster Erasure Coding Small Object Grouping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage cluster methods for permanent storage using erasure codes are inefficient for small objects, particularly the encode-now approach, which requires large storage space due to inefficient chunking and encoding of small objects on long hard disk tracks.
Innovation Solution
A storage cluster system that combines small objects with other objects to form larger chunks, encodes them using erasure codes, and utilizes a non-volatile temporary memory to manage latency and notify sources of permanent storage early, thereby optimizing storage space and throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If small objects are chunked into very small sizes and encoded immediately (encode-now approach), then storage space requirements are reduced, but encoding and storage efficiency deteriorates due to inefficient utilization of long hard disk tracks
Solution Approach 1:
The patent combines multiple small objects in an input buffer into a larger composite object before chunking and encoding. This merging approach creates larger data units that efficiently utilize hard disk track capacity, improving encoding and storage efficiency while still maintaining reduced storage space requirements compared to full replication
Solution Approach 2:
The system performs preliminary buffering of small objects before final encoding and permanent storage. Objects are temporarily held in an input buffer, allowing them to be combined with other small objects to form larger units. This preliminary action enables efficient track utilization without requiring full replication storage space
2Reliability
If replication is used for permanent storage, then object recovery reliability is improved, but storage space requirements increase significantly
Solution Approach 1:
The patent changes the storage parameter from full replication (storing complete object copies) to erasure coding (storing encoded chunks). This parameter change maintains reliable object recovery through the mathematical properties of erasure codes while dramatically reducing storage space requirements by storing only the minimum necessary redundant information
Solution Approach 2:
The system introduces erasure coding as an intermediary process between object reception and permanent storage. Instead of directly replicating objects, the objects are encoded into chunks with redundant information, serving as a mediator that provides both reliability and space efficiency
3Productivity
If small objects are buffered and combined into larger chunks, then encoding efficiency is improved, but device complexity increases due to buffer management requirements
Solution Approach 1:
The input buffer automatically manages small objects through predefined combining logic and latency monitoring. The system self-regulates by detecting when buffered objects have waited long enough and automatically triggering encoding operations, reducing the need for complex external buffer management mechanisms
Data Source
AI summary
Small objects are efficiently stored with erasure codes by combining a small object with other small objects and/or large objects to form a single large object for chunking, and providing early notification of permanent storage to the sources of the objects to prevent small objects from becoming stale while waiting for additional objects to be combined.