Distributed Virtual Array Data Reconstruction via Key-Based Addressing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed data storage systems, managing data availability, reliability, capacity, and performance is challenging due to the complexity of storing data across multiple devices, especially when dealing with failures and the need for efficient reconstruction of lost data, while also aiming to reduce IT costs and increase flexibility.
Innovation Solution
The Distributed Virtual Array (DVA) system addresses these challenges by organizing data into containers and chunks, using a unique naming system and error correction codes, and employing a layout table to distribute chunks across multiple storage devices, enabling efficient reconstruction and load balancing without the need for traditional RAID semantics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is distributed across multiple storage devices using traditional RAID levels, then data reliability and availability are improved through mirroring and error correction, but device complexity and administrative overhead increase due to mapping tables and reconstruction coordination
Solution Approach 1:
The patent segments data into fixed-size chunks and distributes them across multiple storage devices using a hash-based addressing scheme. Each chunk is independently addressable through a simplified key format that eliminates the need for complex mapping tables. This segmentation approach maintains data reliability through distribution while reducing administrative complexity by removing the need for traditional RAID mapping infrastructure.
Solution Approach 2:
The patent introduces a simplified intermediary layer consisting of fixed-size chunks and hash-based keys that mediates between the storage devices and the data reconstruction process. This intermediary structure replaces complex mapping tables with a straightforward hash function, enabling efficient data retrieval and reconstruction without the overhead of traditional RAID management systems.
2Ease of operation
If mapping tables are implemented to track data locations across distributed devices, then data accessibility is improved, but loss of time occurs during table updates and synchronization
Solution Approach 1:
The patent performs preliminary action by pre-computing hash keys for each data chunk during the write operation. These hash keys serve as direct addresses for storage device selection, eliminating the need for runtime table lookups. The chunk addressing scheme is established in advance through the hash function, enabling immediate data retrieval without time-consuming table synchronization.
Solution Approach 2:
The patent replaces the mechanical system of mapping tables and their associated update/synchronization operations with a computational hash-based addressing scheme. The hash function provides a deterministic, O(1) method for determining chunk locations without requiring any table maintenance, thereby eliminating time loss associated with table updates while maintaining excellent data accessibility.
3Reliability
If redundant copies of data are stored across multiple devices, then data availability is improved during failures, but loss of substance occurs in the form of increased storage requirements
Solution Approach 1:
The patent changes the parameter of redundancy from full mirroring to fixed-size chunk distribution with hash-based addressing. Instead of storing complete redundant copies of entire data sets, the system distributes fixed-size chunks across multiple devices, providing fault tolerance through distribution while minimizing storage overhead. This parameter change enables data availability during failures without the substantial storage consumption of traditional mirroring approaches.
Data Source
AI summary
In a Distributed Virtual Array data storage system, data chunks making up data containers are stored in a key order in storage devices according to layouts specified in a layout data structure. When any of the storage devices becomes inaccessible, the storage devices iteratively return, in storage order, their stored chunks. Chunks belonging to the same container are identified and, if that container had a chunk in the inaccessible storage device, are used to reconstruct the inaccessible chunk. The chunks of the container are then stored according to an updated layout. The keys are independent of physical storage locations.