Scan Kernel Input Buffer Adaptation for Database Scan Acceleration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Sequential scan operations in datacenter online analytical processing (OLAP) workloads are bottlenecked by data movement between storage and memory, particularly in database systems like Postgres with complex data formats, which hinder acceleration and efficiency.
Innovation Solution
Implementing a scalable streaming architecture with computational storage smart devices that perform scan and filtering operations near the data storage device, utilizing a scan kernel with adaptable input buffers and preprocessing modules to reduce data movement and enable parallel processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is moved from storage device to host memory for CPU-based scan operations, then scan operations can be performed using CPU instructions, but data movement becomes a bottleneck reducing throughput and increasing latency
Solution Approach 1:
The system segments the scan operation into two parts: data movement is handled by the storage device's internal controller, while the actual scan and filter operations are performed by dedicated hardware circuits within the storage device. This segmentation eliminates the need to move all data to host memory, as only filtered results need to be transferred back.
Solution Approach 2:
The patent introduces an intermediary scanning and filtering device that resides within the storage device's controller. This intermediary component acts as a bridge between the storage medium and the host system, performing preprocessing operations locally before data leaves the storage device, thereby reducing the burden on the host CPU and minimizing data movement.
2Adaptability or versatility
If CPU-based scan operations are used with complex data formats like Postgres, then flexible data processing is achieved, but the complexity of data formats makes hardware acceleration difficult
Solution Approach 1:
The system changes the parameter of data format handling by implementing configurable preprocessing modules that can adapt to different data formats. The hardware is designed with parameterizable structures that can be configured to handle various formats like Postgres, allowing flexibility without requiring completely different hardware designs for each format.
Solution Approach 2:
The scanning and filtering device is designed with universal components that can handle multiple data formats through configurable preprocessing stages. The same hardware architecture can be adapted to work with different formats by adjusting control parameters and preprocessing logic, making the system multi-functional without proportionally increasing complexity.
3Reliability
If all data is moved to host memory for processing, then complete data availability is ensured, but power consumption increases due to excessive data movement
Solution Approach 1:
The system performs preliminary filtering actions within the storage device before data is transferred to the host system. By pre-processing the data and eliminating non-matching records at the source, the amount of data that needs to be moved to host memory is significantly reduced, thereby lowering power consumption while ensuring that all necessary data is still available for processing.
Data Source
AI summary
A method and a memory device are provided. Data is obtained for a scan operation at an input buffer of a scan kernel in the memory device. The input buffer is adaptable to a first mode and a second mode of the scan kernel. Preprocessing of the data from the input buffer is performed to generate preprocessed data. A different type of preprocessing is performed for the first mode and the second mode. The preprocessed data is filtered to generate a filtered result. The filtered result is provided from the scan kernel to a controller of the memory device.