Redundant Data Assignment in Distributed Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Enterprise-class data storage systems face challenges in scalability and cost due to the need for customized, high-cost hardware and lack of efficient fault tolerance, particularly in managing redundant data replication across commodity storage components.
Innovation Solution
The implementation of a data storage system that uses commodity components with decentralized operation, employing redundant data replication and timestamp management to ensure data consistency and availability, allowing for coordinated read and write operations, and dynamic node management to balance load and remove underutilized nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If customized reliable hardware components are used to ensure fault tolerance, then data reliability is improved, but system cost and complexity increase
Solution Approach 1:
The patent implements fault tolerance by creating redundant copies of data across multiple commodity storage devices. Instead of using specialized reliable hardware, the system stores multiple replicas of the same data on standard off-the-shelf components, achieving reliability through duplication rather than through expensive specialized hardware
Solution Approach 2:
The patent makes commodity storage devices perform the dual function of both primary data storage and fault tolerance. The same standard storage components that hold primary data also serve as backup storage through replication, eliminating the need for separate specialized reliable hardware components
2Reliability
If data replication is implemented on commodity storage systems, then fault tolerance is improved, but coordination complexity and synchronization overhead increase
Solution Approach 1:
The patent implements a decentralized architecture where each storage node autonomously manages its own data replicas and makes independent decisions about data placement and replication. Nodes self-organize without requiring complex centralized coordination or synchronization protocols, reducing overall system coordination complexity
Solution Approach 2:
The patent divides the storage system into independent autonomous nodes, each managing its own segment of the replicated data. This segmentation allows nodes to operate independently without requiring complex inter-node coordination, as each node handles its own replication and fault tolerance locally
3Quantity of substance
If a single storage system is expanded to support finite capacity, then initial cost is reduced, but scalability and performance limits are reached
Solution Approach 1:
The patent implements a dynamic storage system where nodes can be freely added or removed from the distributed storage network. The system automatically adapts to changing capacity requirements by dynamically rebalancing data across the available nodes, allowing continuous scalability without fixed system limits
Solution Approach 2:
The patent enables unlimited scalability by allowing data to be replicated across an expanding number of nodes. As the system grows, new nodes automatically receive replicated copies of existing data, providing linear capacity expansion without reaching fixed system limits
Data Source
AI summary
The present invention provides techniques for assignment and layout of redundant data in data storage system. In one aspect, the data storage system stores a number M of replicas of the data. Nodes that have sufficient resources available to accommodate a requirement of data to be assigned to the system are identified. When the number of nodes is greater than M, the data is assigned to M randomly selected nodes from among those identified. The data to be assigned may include a group of data segments and when the number of nodes is less than M, the group is divided to form a group of data segments having a reduced requirement. Nodes are then identified that have sufficient resources available to accommodate the reduced requirement. In other aspects, techniques are providing for adding a new storage device node to a data storage system having a plurality of existing storage device nodes and for removing data from a storage device node in such a data storage system.