Virtual Storage Layer Hybrid Reliability for Key-Value Objects
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data storage systems face inefficiencies in storing large key-value objects due to performance bottlenecks and space limitations, as existing replication and erasure coding methods are either too expensive for large objects or wasteful for small ones, and do not efficiently handle objects of varying sizes.
Innovation Solution
A data storage system employing a virtual storage layer that dynamically classifies objects based on size, applying data replication for small objects and erasure coding for large objects, splitting huge objects into chunks and distributing them across multiple devices, and using a hybrid reliability mechanism to optimize storage and retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data replication scheme is applied to large objects, then data reliability is improved, but storage cost and space overhead increase significantly
Solution Approach 1:
The system applies different data reliability schemes to different data objects based on their size characteristics. Small objects use data replication for fast access, while large objects use erasure coding for space efficiency. This local differentiation resolves the contradiction by matching the reliability mechanism to the specific needs of each data size category.
Solution Approach 2:
The system dynamically changes the reliability parameter (replication factor vs. erasure coding rate) based on the object size parameter. By setting a threshold size, objects below the threshold use replication with higher reliability overhead, while objects above the threshold use erasure coding with lower storage overhead, thus resolving the contradiction between reliability and storage cost.
2Quantity of substance
If erasure coding scheme is applied to small objects, then storage space is optimized, but performance and retrieval efficiency deteriorate
Solution Approach 1:
The system applies erasure coding specifically to large objects that benefit from space optimization, while using data replication for small objects where retrieval efficiency is critical. This localized application resolves the contradiction by avoiding erasure coding overhead for small objects that require fast access.
Solution Approach 2:
The system changes the reliability scheme parameter based on object size. For small objects below the threshold, it uses replication with O(1) access time. For large objects above the threshold, it uses erasure coding with O(n) access time, thus optimizing both storage space and retrieval efficiency for different object sizes.
3Device complexity
If a single data reliability scheme is used for all objects, then system complexity is reduced, but adaptability to different object sizes deteriorates
Solution Approach 1:
The system dynamically selects the appropriate data reliability scheme based on the size of each object. By introducing a size threshold parameter, the system adapts its behavior: using replication for small objects and erasure coding for large objects. This dynamic adaptation resolves the contradiction between system simplicity and adaptability.
Solution Approach 2:
The system segments the data object population into two categories based on size: small objects and large objects. Each segment receives a tailored reliability scheme appropriate to its characteristics. This segmentation resolves the contradiction by allowing different schemes for different segments while maintaining overall system manageability.
4Speed
If data replication is used for all objects, then retrieval speed is improved, but storage overhead increases for large objects
Solution Approach 1:
The system applies data replication locally to only those objects that benefit from fast retrieval (small objects), while using erasure coding for large objects where storage overhead becomes prohibitive. This resolves the contradiction by matching the retrieval-optimized scheme to the right data size category.
Solution Approach 2:
The system changes the storage scheme parameter based on object size threshold. For objects below the threshold, it uses replication providing O(1) retrieval speed. For objects above the threshold, it uses erasure coding to reduce storage overhead, thus balancing retrieval speed and storage overhead based on object characteristics.
Data Source
AI summary
A data storage system includes: a plurality of data storage devices for storing a plurality of objects of a key-value pair; and a virtual storage layer that applies different data reliability schemes including a data replication scheme and an erasure coding scheme based on a size of an object of the plurality of objects. The plurality of objects includes a first object having a first size and a second object having a second size that is larger than the first size. The virtual storage layer classifies the first object as a small object, applies the data replication scheme, and stores the small object across one or more of the plurality of data storage devices. The virtual storage layer classifies the second object as a huge object, splits the huge object into one or more chunks of a same size, applies the erasure encoding scheme, and distributedly stores the one or more chunks across the plurality of data storage devices.