Cloud Query Result Fetching via Direct Object Store Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud-based computation systems often face bottlenecks when handling large job outputs due to the control system acting as a bottleneck, especially when multiple users and jobs are receiving service, leading to inefficiencies in data access and transfer.
Innovation Solution
A system that enables direct and parallel access to output data stored in cloud object stores by determining whether outputs are in-line or cloud-based, allowing for efficient aggregation and metadata generation to facilitate direct client access, thereby avoiding the control system bottleneck.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the control system processes all job outputs centrally, then data consistency and security are maintained, but system performance degrades due to the control system becoming a bottleneck
Solution Approach 1:
The patent segments data access into two paths: control system path for small outputs and direct cloud storage path for large outputs. This segmentation allows the control system to handle only essential metadata and authentication, while large datasets are accessed directly from cloud storage, eliminating the bottleneck and maintaining data consistency through centralized metadata management.
Solution Approach 2:
The patent introduces cloud object storage as an intermediary between the control system and the data requester. The control system acts as a mediator that provides authentication and metadata, while the actual data transfer occurs through the cloud storage system, reducing the control system's processing burden and preventing it from becoming a bottleneck.
2Reliability
If the control system transfers all outputs through centralized processing, then data security is maintained, but transfer efficiency decreases for large files
Solution Approach 1:
The patent segments the data transfer process by size: small outputs go through centralized control system processing for security verification, while large outputs are transferred directly from cloud storage. This segmentation maintains security for critical data while dramatically improving transfer efficiency for large files by eliminating unnecessary centralized processing steps.
Solution Approach 2:
The cloud object storage system serves as an intermediary that handles secure data transfer for large files. The control system maintains security through authentication and metadata management, while the actual file transfer occurs through the cloud storage intermediary, improving transfer efficiency without compromising security.
3Device complexity
If the control system processes all output types uniformly, then system simplicity is maintained, but processing time increases for large outputs
Solution Approach 1:
The patent implements dynamic processing where the system automatically adjusts its behavior based on output size. The control system dynamically routes small outputs through centralized processing and large outputs to direct cloud storage access. This dynamic approach maintains system simplicity through automated routing while significantly reducing processing time for large outputs.
Solution Approach 2:
The patent segments processing based on output size thresholds. The control system maintains simplicity through a single unified architecture that automatically segments data flow based on size, routing small datasets through centralized processing and large datasets to direct cloud access, thereby reducing processing time without adding complex manual intervention.
4Ease of operation
If the control system aggregates all outputs centrally, then data organization is improved, but the control system becomes a bottleneck for multiple concurrent jobs
Solution Approach 1:
The patent segments the aggregation function by data size: small outputs are aggregated through the control system for organization, while large outputs are aggregated directly in cloud storage. This segmentation allows the control system to maintain data organization for small datasets while concurrently handling multiple jobs without becoming a bottleneck, as large outputs are organized directly in the cloud.
Solution Approach 2:
The cloud storage system acts as an intermediary that handles aggregation for large outputs. The control system maintains data organization through metadata management for small outputs, while large outputs are aggregated and organized directly in the cloud intermediary, enabling concurrent job handling without the control system becoming a bottleneck.
Data Source
AI summary
A cloud computation system configured to 1) receive a first request to read a first set of query results stored in a cloud based data storage; 2) transmit a first subset of the first set of query results in response to the first request; 3) transmit a second subset of the first set of query results in response to the first request; 4) receive a second request to read a second set of query results stored in the cloud based data storage; 5) transmit a first subset of the second set of query results in response to the second request. 6) transmit a second subset of the second set of query results in response to the second request.