Serverless Disk Drives via Cloud Blob Buckets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed file systems in cloud computing environments face challenges in managing large data sets with high availability and redundancy, as existing tools do not efficiently handle storage node failures and performance scaling, leading to inefficiencies in data storage and retrieval.
Innovation Solution
The implementation of a system that utilizes block blobs in cloud storage, where write and read requests are managed by determining the appropriate address ranges and associating them with buckets, allowing for efficient data storage and retrieval by employing cache and block storage volumes, and dynamically provisioning or releasing storage resources based on demand.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional distributed file systems are used with built-in tools for managing storage volumes, then data storage capacity is provided, but storage node failures and performance scaling are not efficiently handled
Solution Approach 1:
The patent introduces cloud storage services as an intermediary layer between the distributed file system and physical storage devices. This intermediary handles storage volume management, failure recovery, and scaling operations, resolving the contradiction by providing both reliability through cloud-managed redundancy and productivity through automated operations that reduce manual intervention overhead
Solution Approach 2:
The system implements self-service mechanisms where the cloud storage service automatically manages storage volume provisioning, failure detection, and data recovery without manual intervention. The distributed file system automatically interacts with cloud storage APIs to maintain data availability and handle scaling, improving both reliability and operational efficiency
2Adaptability or versatility
If cloud storage services are used to provide storage capacity, then scalability is improved, but operational differences from physical storage services create management inefficiencies
Solution Approach 1:
The patent creates a universal storage management interface that works with both cloud storage services and traditional physical storage devices. The system abstracts storage-specific operations into a common API layer, allowing the distributed file system to interact with diverse storage backends through standardized commands, thereby improving scalability while reducing management complexity through interface unification
Solution Approach 2:
The system dynamically adjusts storage management parameters based on the type of storage backend being used. When cloud storage is detected, the system automatically modifies volume management parameters, provisioning behaviors, and failure recovery strategies to match cloud service characteristics, enabling scalable operations without manual configuration complexity
3Reliability
If storage resources are statically provisioned for distributed file systems, then data availability is maintained, but storage utilization is not optimized and costs increase
Solution Approach 1:
The patent implements dynamic storage resource provisioning where storage capacity is automatically adjusted based on actual data availability requirements and usage patterns. The system continuously monitors storage utilization metrics and dynamically provisions or releases cloud storage volumes, maintaining data availability through on-demand resource allocation while optimizing utilization by eliminating static over-provisioning
Solution Approach 2:
The system incorporates feedback mechanisms that monitor storage usage patterns, data access frequencies, and availability requirements. This feedback drives automated decisions about storage volume provisioning and deprovisioning, ensuring data availability is maintained through real-time resource adjustment while optimizing storage utilization by releasing unused capacity and preventing premature provisioning
Data Source
AI summary
Embodiments are directed to managing data in a file system that includes a plurality of storage nodes and a plurality of storage volumes such that each storage node may be a compute instance provided by a cloud computing environment (CCE) and each storage volume may be based on blobs provided by the CCE. Write requests that include write data may be provided. A lowest range of addresses in the storage volume may be determined for storing the write data. Buckets may be determined based on the range of addresses and the write data assigned to the buckets. If portions of the buckets remain unassigned, other data may be provided from a cache volume or the blobs. The write data and the other data may be written to the portion of the blobs.