Write Request Reordering for Storage Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems face inefficiencies in scheduling write operations, particularly when dealing with varying data set sizes, leading to potential delays in completing large write operations due to the prioritization of smaller ones in a First-In, First-Out (FIFO) processing system, which can violate time limits and copyright restrictions.
Innovation Solution
Implementing a latency reduction system that reorders write operations based on their size and priority, determining a wait period and start time for each operation to ensure completion within client-defined thresholds, while also optimizing storage entity selection for efficient data and parity distribution across multiple storage entities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data mirroring is used to provide fault tolerance, then reliability is improved, but storage space utilization increases significantly
Solution Approach 1:
The patent uses erasure coding to create fractional copies (parity blocks) of data blocks instead of full mirrors. Each data block is divided into K segments with N-K parity segments, providing fault tolerance while using only N/K storage space ratio instead of 2:1 or 3:1 ratios required by traditional mirroring
Solution Approach 2:
The system changes the fault tolerance parameter from binary (mirrored/not mirrored) to a continuous spectrum by adjusting K and N values in the erasure coding scheme, allowing flexible trade-off between reliability and storage efficiency based on specific requirements
2Productivity
If smaller write operations are scheduled first in FIFO queue, then queue processing efficiency is improved, but large write operations may exceed completion time limits
Solution Approach 1:
The patent implements dynamic prioritization where the scheduler adjusts write operation priorities based on real-time factors including data size, deadline requirements, and queue state. Large writes with tight deadlines can be elevated in priority despite arriving later, making the scheduling policy adaptive rather than static
Solution Approach 2:
The system changes the scheduling parameter from simple FIFO order to a multi-dimensional priority metric that incorporates write size, deadline urgency, and completion probability, allowing the scheduler to optimize for both throughput and individual operation success rates
3Reliability
If multiple copies of data are stored for fault tolerance, then reliability is improved, but compliance with copyright restrictions deteriorates
Solution Approach 1:
The patent segments data into blocks and further divides each block into K segments that are distributed across N storage entities. This segmentation approach creates a single logical instance of data that is physically distributed, satisfying copyright's single-instance requirement while providing fault tolerance through redundant segments
Data Source
AI summary
In some implementations, a method includes, at a latency reduction system configured to reduce latency in writing data to one or more storage entities that are each configured to store data on an object-basis, receiving a scheduling request associated with a write operation. The method also includes determining a wait period for the write operation, where the wait period is less than or equal to the difference between a client-defined operation threshold and an operation completion time of the write operation and determining an operation start time for the write operation, based in part on the wait period.