Storage Device Adapter for Accelerating Database Temporary Table Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database queries often result in temporary tables that are too large to fit in host memory, leading to increased latency due to frequent I/O operations between host and secondary storage, which existing near-storage solutions fail to adequately address.
Innovation Solution
An accelerator adapter is used to minimize the back-and-forth data transfer between host and device by delegating tasks such as merging, splitting, and histogram operations to a storage device like an SSD, utilizing Peer-to-Peer technology and reducing the need for large amounts of device DRAM, thus improving query response times.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If temporary tables are stored in secondary storage due to size constraints, then memory capacity requirements are reduced, but I/O operations increase and processing time increases
Solution Approach 1:
The storage device acts as an intermediary by performing merge, split, and histogram operations directly on temporary tables stored in secondary storage, eliminating the need for frequent I/O operations between host and storage device. This mediator approach allows large temporary tables to be processed efficiently without increasing I/O latency.
Solution Approach 2:
The storage device performs preliminary processing operations (merge, split, histogram) on temporary tables before they need to be accessed by the host. By preparing and organizing data in advance on the storage device, the system reduces subsequent I/O operations and accelerates query processing.
2Adaptability or versatility
If data is frequently transferred between host and secondary storage for processing, then processing flexibility is maintained, but I/O exchange increases and performance decreases
Solution Approach 1:
The storage device serves as an intermediary that performs processing operations locally on temporary tables, reducing the need for frequent data transfers between host and storage. This maintains processing flexibility while significantly improving query processing speed by minimizing I/O exchange.
Solution Approach 2:
The storage device performs self-service processing by executing merge, split, and histogram operations directly on temporary tables stored in its memory, without requiring constant host intervention or data transfer. This self-service capability maintains processing flexibility while accelerating query execution.
3Ease of operation
If host processor performs all temporary table operations, then control and data access are simplified, but processor load increases and bottleneck occurs
Solution Approach 1:
The storage device acts as an intermediary that handles temporary table operations locally, reducing the processing burden on the host processor. This maintains ease of data access for the host while eliminating the processor bottleneck that would occur if all operations were performed on the host.
Solution Approach 2:
The system extracts temporary table processing operations from the host processor and relocates them to the storage device. By taking out these computationally intensive operations from the host, the system reduces processor load and eliminates the bottleneck while maintaining simplified data access mechanisms.
Data Source
AI summary
A system is described. The system may include a storage device, which may include storage for data and a controller to write the data to the storage and to read the data from the storage. The storage device may also include an accelerator implementing at least one operation on files stored on the storage device. These operations may be drawn from a set including a merge operation and a split operation.