Cloud Storage NVRAM Buffering for Write Latency and Reliability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional storage systems face inefficiencies in managing data storage and retrieval operations, particularly in cloud-based environments, where latency and reliability issues arise due to direct writing to storage drives, leading to unnecessary write operations and potential data loss during power failures.
Innovation Solution
The implementation of a cloud-based storage system that utilizes non-volatile random access memory (NVRAM) as a buffer for data destined to be written to storage drives, offloading device management responsibilities from storage drives, and employing dual storage array controllers with primary and secondary status to ensure data integrity and availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is directly written to storage drives, then data persistence is achieved, but write latency increases and unnecessary write operations occur
Solution Approach 1:
The patent introduces NVRAM as an intermediary buffer between the host system and storage drives. Data is first written to the NVRAM buffer, which provides fast write access, and then asynchronously transferred to the storage drives. This mediator layer resolves the contradiction by providing both fast writes (reducing latency) and persistent storage (ensuring data safety).
Solution Approach 2:
The system performs preliminary writing of data to the NVRAM buffer before final storage to the drives. This preliminary action allows the system to acknowledge write operations quickly to the host while the data is still being safely transferred and committed to the storage drives in the background, thus reducing perceived write latency while maintaining data persistence.
2Reliability
If data is directly written to storage drives, then data availability is ensured, but system reliability decreases during power failures
Solution Approach 1:
The NVRAM buffer acts as a protective cushion that absorbs and protects against power failure impacts. Data is first committed to the non-volatile NVRAM buffer, which preserves data even without power, and then gradually transferred to storage drives. This beforehand cushioning ensures that if a power failure occurs during the transfer process, no data is lost because it is already safely stored in the NVRAM buffer.
3Productivity
If storage drives manage their own device responsibilities, then device autonomy is maintained, but processing overhead increases and performance decreases
Solution Approach 1:
The patent extracts the device management responsibilities from the storage drives themselves and consolidates them into dedicated storage array controllers. The controllers handle metadata management, I/O scheduling, and data movement coordination, while the storage drives focus solely on data storage and retrieval. This extraction reduces the processing overhead on individual drives and improves overall I/O operation efficiency.
Solution Approach 2:
The storage array controllers are designed as universal management units that handle multiple functions including metadata management, I/O request scheduling, data transfer coordination, and buffer management. By consolidating these diverse management responsibilities into multi-functional controllers, the system achieves better resource utilization and reduced complexity compared to having each drive manage its own operations.
4Reliability
If single controller architecture is used, then device simplicity is maintained, but system reliability decreases during controller failures
Solution Approach 1:
The patent implements a dual controller architecture where each controller is specialized with specific responsibilities - one controller is designated as primary for normal operations while the other serves as secondary/standby. This local quality differentiation allows the system to maintain simplicity in normal operation while providing redundancy for failure scenarios. The primary controller handles all I/O operations, while the secondary controller remains ready to take over, ensuring high availability without requiring complex active-active coordination.
Data Source
AI summary
Servicing I/O operations in a cloud-based storage system, including: receiving, by the cloud-based storage system, a request to write data to the cloud-based storage system; storing, in solid-state storage of the cloud-based storage system, the data; storing, in object storage of the cloud-based storage system, the data; detecting that at least some portion of the solid-state storage of the cloud-based storage system has become unavailable; identifying data that was stored in the portion of the solid-state storage of the cloud-based storage system that has become unavailable; retrieving, from object storage of the cloud-based storage system, the data that was stored in the portion of the solid-state storage of the cloud-based storage system that has become unavailable; and storing, in solid-state storage of the cloud-based storage system, the retrieved data.