Key-Value Shard Creation With Coordinator Mapping and Copy-On-Write
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face challenges in scalability and performance due to their proprietary nature, making upgrades costly and time-consuming, and they struggle to adapt to emerging technologies like AI and machine learning.
Innovation Solution
A scalable data storage software suite that enables key-value shard creation and management, allowing for partitioning into coordinator and value shards with redirected write operations and copy-on-write cloning, supporting flexible and efficient data storage on various hardware platforms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If proprietary hardware and software clusters are used, then system integration is tight and performance is optimized, but upgrading components becomes costly and time-consuming
Solution Approach 1:
The system separates storage software from hardware infrastructure, dividing the storage system into independent software layers that can be upgraded without replacing entire hardware clusters. This segmentation allows individual software components to be updated while maintaining hardware compatibility.
Solution Approach 2:
The storage software is designed to run on multiple hardware platforms and configurations, making it universal rather than proprietary to specific hardware. This allows the same software to operate across different hardware generations and vendors, enabling flexible upgrades.
2Stability of the object's composition
If outdated purpose-built hardware is used, then system stability is maintained, but system performance becomes sub-par
Solution Approach 1:
The system enables dynamic hardware updates by allowing software to adapt to new hardware configurations without requiring complete system reconfiguration. This dynamic approach allows performance optimization through hardware upgrades while maintaining operational stability through software continuity.
3Productivity
If entire infrastructure is replaced to adopt latest hardware, then performance and scalability improve, but costs increase
Solution Approach 1:
The architecture segments the storage system into replaceable software components and persistent hardware infrastructure, allowing incremental adoption of new hardware without replacing the entire system. This reduces implementation costs by enabling phased upgrades.
Solution Approach 2:
The system uses data replication and shard copying mechanisms that allow data to be migrated to new hardware without requiring complete system replacement. This enables performance improvement through hardware upgrades while maintaining data accessibility and reducing migration costs.
Data Source
AI summary
Methods and systems for key-value shard creation and management in a key-value store (319). A method includes the steps of partitioning the key-value store (319) including a plurality of keyspaces (364A, 364B) into a plurality of shards including a coordinator shard (362), receiving a first shard assignment value for a first key-value pair from a key-value store client, the key-value store client having access to the key-value store, creating a first shard (321A) using the first shard assignment value, the first shard including a data structure, assigning the first key-value pair to the first shard (321a) based on the first shard assignment value, updating the coordinator shard (362) to reference to the first shard (321A), writing the first shard (321A) to a storage device, and storing modifications to the first shard (321A) in a modified version of the first shard (321A) while the first shard (321A) remains unmodified.