Cloud File Descriptors for Direct Object Storage Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing cloud storage systems face bottlenecks during file uploads and downloads due to the presence of a cloud connector in the data path, especially when the connector is not co-located with the object storage system, leading to inefficiencies in handling concurrent clients and storage replication across multiple locations.
Innovation Solution
The implementation of a cloud file descriptor system that allows clients to directly read from or write to object storage systems without a cloud connector in the data path, using download or upload tokens and encryption keys, which are encoded as signed URLs to authenticate and authorize access, enabling efficient data retrieval and storage without additional authentication steps.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a cloud connector is used to manage file uploads and downloads in a cloud storage system, then centralized control and security management are improved, but network traffic efficiency deteriorates and the system becomes a bottleneck for concurrent clients
Solution Approach 1:
The system segments the cloud storage interface into multiple distributed connectors that can independently handle client requests. Each connector operates autonomously to upload and download files directly between clients and object storage systems, eliminating the single-point bottleneck while maintaining centralized security policies through token-based authentication.
Solution Approach 2:
The system introduces cloud file descriptors as intermediary tokens that enable direct peer-to-peer file transfers between clients and object storage systems. These descriptors contain authentication credentials that allow clients to bypass the cloud connector during actual data transmission, while the connector retains control for authorization and monitoring purposes.
2Reliability
If the cloud connector is placed inside a secure corporate private network for security reasons, then security management is improved, but network accessibility deteriorates for Internet-based clients
Solution Approach 1:
The system extracts the authentication and authorization functions from the cloud connector, embedding them directly into cloud file descriptors that are issued to clients. This allows clients to authenticate securely through the connector once, then perform direct transfers with the embedded credentials, eliminating the need for continuous connector involvement and improving accessibility.
Solution Approach 2:
The system performs preliminary authentication and authorization actions through the cloud connector before actual file transfers. The connector issues cloud file descriptors containing valid credentials in advance, allowing clients to perform direct transfers without needing continuous network access to the secure corporate environment.
3Productivity
If files are stored in a segmented and distributed fashion across multiple object storage systems, then storage scalability and reliability are improved, but data retrieval complexity increases
Solution Approach 1:
The system creates a universal cloud file descriptor format that can reference objects across multiple different object storage systems. The descriptor contains standardized authentication credentials and object metadata that work uniformly regardless of which underlying storage system the objects reside in, simplifying the retrieval process despite distributed storage complexity.
Data Source
AI summary
A method and system method for reading a file from a cloud storage service are provided. The method includes receiving, from a client device, a request to read at least a portion of a file from a cloud storage service, wherein the cloud storage service is associated with at least one object storage system; and sending a cloud file descriptor to the client device, wherein the cloud file descriptor includes a plurality of download tokens utilized to retrieve objects constituting the requested at least a portion of the file from the at least one object storage system.