Cloud Query Platform Inversion for Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional approaches to accessing cloud storage services face challenges with network latency and transmission volume when dealing with large volumes of data, particularly in unstructured and semi-structured databases, where strong typing constraints and high computational demands hinder efficient data processing.
Innovation Solution
Implementing a cloud-based query platform that instantiates virtual machines on the cloud storage service, allowing query logic to be executed directly on the data, thereby reducing the need to transport large datasets and leveraging cloud computing resources for efficient data processing and query execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is transported from cloud store to local application for processing, then data can be accessed and processed, but network latency and transmission volume increase substantially
Solution Approach 1:
Instead of transporting data from cloud storage to local application (conventional approach), the patent inverts the approach by bringing the database and query processing capabilities to the data location in the cloud. Virtual machines are instantiated on the cloud storage service, allowing query logic to be executed directly on the stored data without network transport, thereby eliminating network latency and transmission bottlenecks.
2Productivity
If full intake processing is performed on all data collections, then query access efficiency is improved, but computational burden and processing time increase
Solution Approach 1:
The patent implements selective intake processing where only data collections with high query traffic or frequent access patterns undergo full columnar storage transformation. Collections with sparse or indefinite usage patterns are maintained in their original format with lighter cataloging or indexing. This partial application of intensive processing optimizes query efficiency for frequently accessed data while avoiding unnecessary computational burden on seldom-accessed collections.
3Productivity
If virtual machines are instantiated on cloud storage service, then query processing efficiency improves by eliminating data transport, but device complexity increases
Solution Approach 1:
The patent leverages the existing cloud storage service's virtual machine infrastructure to perform multiple functions: data storage, query processing, and computational analysis. By utilizing the cloud provider's existing VM capabilities, the system achieves complex query processing functionality without adding significant infrastructure complexity, as the virtual machines serve both as compute resources and as the execution environment for query logic.
Data Source
AI summary
A query server identifies data collections of interest in a cloud store, and categorizes the collections based on an intended usage. Depending on the intended usage, the categorized data may be cataloged, indexed, or undergo a full intake into a column store. In a database of large data collections, some collections may experience sparse or indefinite usage. Cataloging or indexing position the collections for subsequent query access, but defers the computational burden. The full intake performs a columnar shredding of the collection for facilitating eminent and regular query access. Upon invocation of query activity, an instantiation of virtual machines provided by the cloud store vendor implements query logic, such that the VMs launch in conjunction with the cloud store having the collections. Collections therefore incur processing based on their expected usage-full intake for high query traffic collections, and reduced cataloging for maintaining accessibility of collections of indefinite query interest.