Object Storage Staging Queues for High-Rate Bypass Writes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current storage systems incur substantial expense and delay when writing large amounts of data to performance storage tiers before migrating to capacity storage tiers, and existing mechanisms struggle to handle faster client write rates.
Innovation Solution
Implement a bypass mechanism that allows direct assignment of data to capacity storage tiers, temporarily staging it in a transfer data structure within the performance tier, and mapping it to the capacity tier location for client reference.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is written to performance storage tier before migrating to capacity storage tier, then data can be stored in object based storage, but write performance deteriorates and storage expense increases due to filling up performance storage
Solution Approach 1:
The patent introduces a transfer data structure as an intermediary component between the performance storage tier and capacity storage tier. This transfer data structure serves as a buffer that temporarily holds data during the migration process, allowing data to be written directly to the capacity storage tier without first occupying performance storage capacity. The transfer data structure mediates the data flow and enables efficient data movement while preserving write performance.
Solution Approach 2:
The patent segments the storage system into distinct functional components: performance storage tier for high-speed operations, capacity storage tier for long-term storage, and a transfer data structure for data movement. By segmenting the data path and using a dedicated transfer mechanism, the system avoids the bottleneck of writing large amounts of data through the performance storage tier, thereby maintaining write performance while enabling efficient data ingestion into object based storage.
2Productivity
If data is written directly to capacity storage tier, then write performance improves, but data management complexity increases
Solution Approach 1:
The transfer data structure acts as a managed intermediary that simplifies direct writing to capacity storage. Instead of requiring complex application-layer logic to handle direct-to-capacity writes, the transfer data structure provides a standardized interface that manages the data movement process automatically. This reduces data management complexity while enabling improved write performance through direct capacity tier access.
3Productivity
If faster client write rates are handled, then productivity improves, but existing mechanisms fail to keep up with the write rate
Solution Approach 1:
The transfer data structure enables continuous data ingestion at high write rates by providing an always-available buffer between the client write operations and the capacity storage tier. Data can be continuously written to the transfer data structure at high speeds without blocking, and the migration to capacity storage proceeds continuously in the background. This maintains write rate productivity while ensuring migration reliability through the persistent transfer mechanism.
Data Source
AI summary
A method, computing device, and non-transitory machine-readable medium for managing incoming/outgoing flow through a staging area. A request to write data to an object based storage is received. A determination is made that the data is to be first staged within a transfer data structure. The transfer data structure is in a first storage tier and the object based storage is in a second storage tier. A determination is made that an amount of storage space used in the transfer data structure exceeds a start throttle threshold. In response to the determination that the amount of storage space exceeds the start throttle threshold, the request is sent into a queue selected from a set of throttling queues, wherein requests in the queue are handled in a first in, first out (FIFO) order. The request in the queue is processed based on the FIFO order and a dynamic throttle rate.