Storage Node Data Block Modification for Database Query Acceleration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems face challenges in reducing query execution time due to inefficient data transfer between database nodes and storage nodes, which increases network traffic and processing load on compute nodes.
Innovation Solution
The proposed solution involves a storage node applying evaluation criteria from a database statement to data blocks, modifying them by reshaping and updating block headers, thereby reducing the amount of data transferred and processing required at compute nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If storage nodes pass targeted data blocks from storage to compute nodes for processing the entirety of a database statement, then compute nodes can process complete database statements, but network traffic increases and query execution time increases
Solution Approach 1:
The storage node performs preliminary actions by evaluating predicates and modifying data blocks before transferring them to compute nodes. This includes applying evaluation criteria to data blocks, reshaping them, and modifying block headers in advance, so that compute nodes receive pre-processed data requiring less processing
Solution Approach 2:
The storage node extracts only the necessary portions of data blocks that meet the evaluation criteria. By filtering and selecting specific data blocks that satisfy predicate conditions at the storage node, the system transfers only relevant data to compute nodes, reducing overall data transfer volume
2Quantity of substance
If storage nodes apply evaluation criteria and modify data blocks before transfer, then data transfer volume decreases, but storage node processing complexity increases
Solution Approach 1:
The storage node performs self-service by autonomously evaluating predicates and modifying its own data blocks before transfer. The storage node uses the evaluation criteria from database statements to independently determine which data blocks to transfer and how to modify them, reducing reliance on compute nodes for initial filtering
Solution Approach 2:
The storage node acts as an intermediary between the database statement requirements and the compute node processing needs. It mediates by transforming raw data blocks into optimized formats that meet both storage efficiency requirements and query processing requirements, using evaluation criteria as the basis for transformation
3Productivity
If compute nodes receive and process the entirety of data blocks, then complete database statements can be processed, but processing load on compute nodes increases
Solution Approach 1:
The storage node performs preliminary data block evaluation and modification before transfer, so that compute nodes receive pre-filtered and optimized data blocks. This preliminary processing at the storage node reduces the amount of work compute nodes must perform, lowering their processing load and energy consumption
Data Source
AI summary
Techniques for accelerating query execution by optimizing data transfer between storage nodes and database nodes are provided. In one technique, a compute node receives a database statement and transmits a set of one or more selection criteria associated with the database statement to a storage node. Based on the database statement, the storage node retrieves a set of data blocks from storage. Each data block comprises multiple rows of an index-organized table (IOT), each row comprising a key section and a non-key section. The storage node applies the set of selection criteria to a data block, resulting in a modified data block. The storage node generates a modified header data for the modified data block and transmits the modified data block to the compute node.