Virtual Storage Block Address Tracking for Read Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage systems, especially in large-scale clusters with multiple data centers or available zones, network congestion and subhealth storage servers lead to increased latency and decreased data write performance due to the need for strong consistency through redundancy policies like multi-copy or erasure codes, which can result in read amplification issues.
Innovation Solution
A method and apparatus for a distributed storage system that manages storage using virtual storage blocks, where successful continuous storage address ranges are recorded after each write operation, allowing for direct reading of successfully stored slices without needing additional parity reads, and enabling tolerant handling of slow nodes by considering data successfully written if a preset quantity of slices is stored successfully.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If strong consistency is ensured through redundancy policies (multi-copy or erasure code), then data reliability is improved, but data write performance deteriorates due to network congestion and slow nodes
Solution Approach 1:
The patent applies partial action by requiring only a preset quantity of slices to be stored successfully rather than all copies, allowing the system to tolerate slow nodes and network congestion while maintaining data reliability through the redundancy policy
Solution Approach 2:
The patent segments data into multiple slices that are stored independently across different storage devices, allowing parallel write operations and enabling the system to continue accepting write requests even when some storage devices are slow or congested
2Stability of the object's composition
If all copies must be successfully stored before considering data written, then data consistency is improved, but write latency increases
Solution Approach 1:
The patent uses partial action by considering data successfully written when a preset quantity of slices are stored, rather than requiring all copies to be stored, thereby reducing write latency while maintaining data consistency through the redundancy policy
Solution Approach 2:
The patent implements feedback by tracking the storage status of each slice and notifying the client when the preset quantity of slices have been successfully stored, allowing the client to proceed without waiting for all copies to be stored
3Reliability
If multiple slices are read for parity calculation, then data read reliability is improved, but read amplification increases system resource consumption
Solution Approach 1:
The patent applies preliminary action by pre-calculating and storing parity information when data is written, so that during read operations the parity is already available and no additional slices need to be read for parity calculation, reducing read amplification and system resource consumption
Data Source
AI summary
In a data read/write method, a storage server receives a write request of a client and performs storage. Each write request carries a to-be-written slice, an ID of a first storage device, and a virtual storage address of a first virtual storage block. If storage is performed continuously successfully from a start address within virtual storage space of a virtual storage block in the storage device, a successful continuous storage address range is recorded. For each storage device, all data within the successful continuous storage address range is successfully stored data. When receiving a read request of a client for an address segment within the address range, the storage server may directly return data that needs to be read to the client.