Persistent Database Result Set Chunking and Cloud Offloading
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage and retrieval systems are inefficient in handling large result sets, leading to slow query execution, resource bottlenecks, and limitations in result set consumption, such as synchronous consumption, inability to scroll through results, and lack of auditing and sharing capabilities.
Innovation Solution
The system leverages cloud infrastructure to offload large result sets into chunks stored in cloud storage, using metadata to enable asynchronous consumption and parallel processing, allowing clients to access chunks directly and decoupling query execution from result set consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If the entire result set is sent back from the database server to the client in one computer network pipe, then the query execution is simple, but the query and retrieval of the result set becomes slow
Solution Approach 1:
The patent divides the large result set into multiple chunks that are sent through separate computer network pipes in parallel. Each chunk is transmitted independently, allowing simultaneous data flow through multiple network channels, which significantly increases the overall retrieval speed compared to sending the entire result set through a single pipe
Solution Approach 2:
The patent transitions from a single-dimensional sequential data transmission model to a multi-dimensional parallel transmission model by utilizing multiple network pipes simultaneously. This dimensional change in the data transmission approach enables concurrent transfer of data chunks, resolving the speed limitation of single-pipe transmission while maintaining execution simplicity
2Quantity of substance
If the result set is produced sequentially, then the database server uses fewer resources, but the result set delivery is slow
Solution Approach 1:
The patent segments the result set into multiple chunks that can be produced and transmitted in parallel rather than sequentially. This segmentation enables the database server to utilize multiple processing threads or workers to generate different chunks simultaneously, increasing delivery speed while distributing resource consumption across multiple processing units
Solution Approach 2:
The patent implements continuous parallel production of result set chunks, where multiple chunks are being generated and transmitted simultaneously without idle periods. This continuous parallel action maximizes the utilization of database server resources throughout the query execution process, eliminating the downtime that occurs in sequential processing between completion of one chunk and start of the next
3Reliability
If database server resources are held to support delivery of a large result set, then the result set can be delivered complete, but the database server cannot free up resources to run new queries
Solution Approach 1:
The patent extracts the result set chunks from the database server's primary processing flow and transmits them through dedicated parallel network pipes. This extraction allows the database server to release resources associated with each chunk after transmission begins, while the parallel pipe infrastructure ensures continuous delivery without requiring resources to be held until complete consumption by the client
Solution Approach 2:
The patent introduces parallel network pipes as intermediary transmission channels between the database server and the client. These intermediary pipes buffer and forward result set chunks independently, allowing the database server to decouple from the client's consumption rate and free resources for new queries while maintaining reliable delivery through the intermediary pipe infrastructure
4Device complexity
If the client must consume the result set from the same machine and client where the query is submitted, then the system is simple, but the user cannot go back to a result set after it has been produced and cannot share results among different clients
Solution Approach 1:
The patent creates copies of the result set chunks that are transmitted through parallel pipes and made available to multiple clients. Instead of requiring the original client to consume the entire result set sequentially, the system generates and distributes copies of the data through the parallel pipe infrastructure, enabling different clients to access and consume the same result set independently
Solution Approach 2:
The patent transforms the result set consumption model from a single-client sequential model to a multi-client parallel model by adding the dimension of concurrent access through multiple network pipes. This dimensional change enables result sets to be consumed by multiple clients simultaneously from different machines, providing backwards access and sharing capabilities while maintaining the underlying data structure
Data Source
AI summary
A method, system, and apparatus for managing a persistent query result set are disclosed. A resource manager is disclosed that receives a query from a client over a computer network. A computer database server is disclosed that executes the query and generates a result set based on and in response to the received query. An execution platform is disclosed that divides the result set into a plurality of chunks, wherein each chunk comprises a portion of the generated result set. The resource manager adds metadata to each of the plurality of chunks. A first chunk of the plurality of chunks comprises metadata pointing to the plurality of chunks. The resource manager delivers at least the first chunk of the plurality of chunks to the client in response to the query. The execution platform offloads the remaining plurality of chunks to cloud storage in parallel.