Local Object Transport Proxy Using Presigned Direct Data Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-performance data centers face performance bottlenecks in moving data to and from acceleration hardware due to the overhead introduced by object storage gateways, which reduce data transfer speed and introduce latency.
Innovation Solution
Implementing local object transport accelerator proxies on each computing node to bypass object storage gateways by using presigned requests for direct data access to and from object repositories, maintaining a cluster-wide cache, and asynchronously transferring data between availability zones.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If object storage gateways are used for data access, then authentication and authorization are performed, but data transfer speed is reduced and latency is introduced
Solution Approach 1:
The system performs authentication and authorization in advance by generating presigned requests before actual data transfer. The object storage gateway authenticates the computing node and creates time-limited access URLs that allow direct data transfer without requiring gateway mediation during the actual transfer, thus resolving the contradiction between security and speed
Solution Approach 2:
The data transfer process is segmented into two phases: (1) authentication phase where the gateway verifies credentials and generates presigned requests, and (2) data transfer phase where data flows directly between storage and computing nodes using the presigned requests. This segmentation allows security operations to be separated from data transfer operations, eliminating gateway as a bottleneck during transfer
2Reliability
If data is transferred through object storage gateways, then data access control is maintained, but transfer latency increases
Solution Approach 1:
The presigned request acts as an intermediary mechanism that bridges security requirements and direct data transfer. Instead of the gateway continuously mediating data flow, it issues time-limited access credentials that enable direct transfer while maintaining access control. The presigned request URL serves as a self-contained authorization token that bypasses the need for continuous gateway involvement
Solution Approach 2:
The authentication and authorization functions are extracted from the data transfer path. The gateway performs these functions once to generate presigned requests, then removes itself from the critical data transfer path. This extraction allows data transfer to proceed without gateway overhead while access control is maintained through the presigned credentials
3Productivity
If multiple clusters access the same object repository, then resource utilization is improved, but security risks increase for untrusted clusters
Solution Approach 1:
The object storage gateway performs preliminary authentication and authorization checks before issuing presigned requests to clusters. Each cluster must present valid credentials to obtain access tokens, and the gateway can configure different access policies for different clusters. This preliminary security verification enables multiple clusters including untrusted ones to access the repository safely
Solution Approach 2:
The system creates isolated access contexts for each cluster through presigned requests. Each cluster receives its own set of time-limited access credentials that are specific to its authorized operations. This copying of access control mechanisms to each cluster individually allows secure multi-cluster access without cross-contamination of security contexts
Data Source
AI summary
Methods and systems, including computer-readable media, are described for speeding up data transfer between an object repository and a hardware accelerator. The methods include receiving a request from a hardware accelerator unit to retrieve a data object stored in an object repository. The methods include providing a request to an object storage gateway for a presigned access location and retrieving the data object from the object repository using the presigned access location.