Storage-side scanning for non-native data formats
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database management systems (DBMS) face inefficiencies in data retrieval due to the need to transfer entire data blocks, even though only specific columns are required for query execution, leading to unnecessary data transfer and processing.
Innovation Solution
Implementing storage-side scanning for external tables, which allows a distributed data access system to project and filter columns within the storage system, enabling efficient access and processing of data without the need for entire data block retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the DBMS retrieves entire data blocks from storage for query execution, then complete data availability is ensured, but data transfer overhead and processing time increase significantly
Solution Approach 1:
The patent extracts and applies filtering and projection operations at the storage system level before data is transferred to the DBMS. This removes unnecessary data (columns and rows) from the transfer pipeline, keeping only the specific data needed for query execution, thereby reducing transfer overhead while ensuring data availability for the required operations
Solution Approach 2:
The storage system performs filtering and projection operations preliminarily before the DBMS receives the data. By pre-processing the data blocks to remove irrelevant columns and filter rows based on query conditions, the system prepares only the necessary data for transfer, reducing subsequent processing time at the DBMS while maintaining data availability
2Reliability
If the DBMS transfers and processes entire data blocks, then data completeness is maintained, but data transfer overhead increases
Solution Approach 1:
The storage system extracts and removes unnecessary data elements (specific columns and rows) before transfer to the DBMS. By applying projection to select only required columns and filtering to keep only relevant rows, the system maintains data completeness for the query needs while significantly reducing the volume of data transferred, thereby lowering transfer overhead and energy consumption
Solution Approach 2:
The storage system performs data reduction operations preliminarily before the transfer process. By pre-filtering rows and pre-projecting columns at the storage level, the system prepares compact, query-relevant data subsets that maintain completeness for the intended operations while minimizing the energy required for data transfer across the system boundary
3Productivity
If storage-side scanning is implemented for external tables, then query execution efficiency improves, but system complexity increases
Solution Approach 1:
The storage system is enhanced to perform scanning, filtering, and projection operations autonomously without requiring full data retrieval to the DBMS. This self-service capability at the storage level enables the storage subsystem to independently prepare query-relevant data, improving execution efficiency while distributing complexity to where the data resides rather than centralizing it in the DBMS
Solution Approach 2:
The system architecture is segmented into distinct functional layers: the storage system handles data scanning, filtering, and projection, while the DBMS focuses on query logic and result processing. This segmentation allows each component to specialize in specific operations, improving overall efficiency while organizing complexity into manageable, separated functional modules
Data Source
AI summary
A storage system communicatively coupled to a database management system (DBMS performs storage-side scanning of data sources that are not stored in native database storage format of the DBMS. Data sources for external tables are accessible in a storage system referred to as a distributed data access system (DDAS), e.g. a Hadoop Distributed File System. To execute a query that references an external table, a DBMS first generates an execution plan. The DDAS supplies the DBMS with information that specifies each portion of the data source, and specifies which data node to use to access the portion. The DBMS sends a request for each portion to the respective data node, requesting that the data node generate rows from data in the portion. The request may specify scanning criteria, specifying one or more columns to project and/or filter on, and code modules for the data node to execute to generate records.