Front-End Node Microquery Execution for Distributed Storage Bandwidth Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Efficiently running queries on binary objects stored across multiple storage devices in cloud storage systems is challenging due to high network bandwidth consumption and unnecessary data transfer, as existing methods require transferring all blocks to a common location for filtration, which wastes resources.
Innovation Solution
Implementing data filtration at the data nodes in a distributed storage system, where a front-end node receives a query, identifies data nodes, generates microqueries with predicates, and executes them locally at each data node, returning only relevant data, thereby reducing unnecessary data transfer and resource consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If all blocks are transferred to a common location for query execution, then query processing can be performed centrally, but network bandwidth consumption increases significantly and unnecessary data transfer occurs
Solution Approach 1:
Instead of transferring all data blocks to a central location for query execution, the patent inverts the approach by bringing the query execution capability to the data nodes themselves. Microqueries are generated and executed locally at each data node, eliminating the need to transfer unnecessary data across the network while still achieving complete query processing.
Solution Approach 2:
The patent segments the query processing task into multiple microqueries that are distributed across different data nodes. Each data node executes its own microqueries locally, processing only the relevant blocks of data needed to answer the original query, thereby reducing network bandwidth consumption while maintaining query effectiveness.
2Measurement precision
If data is filtered at the front-end node after transfer, then complete query processing is achieved, but unnecessary data has already been transferred consuming network resources
Solution Approach 1:
The patent applies preliminary action by generating and executing microqueries at the data nodes before transferring data to the front-end node. This preliminary filtering at the source ensures that only relevant data blocks are transferred across the network, eliminating the waste of transferring unnecessary data that would later be filtered out at the front-end.
Solution Approach 2:
The patent introduces microqueries as an intermediary mechanism between the original query and the data blocks. These microqueries act as filters that identify which specific blocks need to be transferred, serving as a mediator that prevents unnecessary data transfer while ensuring complete query processing at the front-end node.
3Productivity
If blocks are distributed across multiple physical devices, then storage scalability is improved, but querying data spanning multiple devices becomes more complex
Solution Approach 1:
The patent segments the query execution process into independent microqueries that can be executed in parallel at different data nodes. Each microquery operates on a specific block of data locally, eliminating the complexity of coordinating query execution across distributed devices while maintaining storage scalability.
Solution Approach 2:
Each data node autonomously executes its own microqueries and determines which blocks need to be transferred, without requiring centralized coordination for each query operation. This self-service approach simplifies the overall system complexity while enabling efficient querying across distributed storage devices.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Examples are disclosed that relate to querying data in a distributed storage system. One example provides, at a front-end node, a method comprising receiving a query regarding data stored in an object, obtaining information identifying data nodes at which blocks of the object are stored, sending, to a first data node, a query regarding data in a first block, and receiving results for the query regarding the data in the first block comprising an incomplete record. The method further comprises sending, to a second data node, a query regarding data in a second block comprising the incomplete record, the second block being stored on a different physical device than the first block, and receiving results for the query regarding the data in the second block. The method further comprises outputting the results for the queries regarding the data in the first block and the data in the second block.