Conditional Column-Store Transcoding for Predicate Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current storage devices, such as Solid State Drives (SSDs), face challenges in efficiently transferring data to host processors due to bandwidth limitations and the need to process compressed or encoded data, which can lead to increased data transmission and processing times.
Innovation Solution
Implementing a transcoder within the storage device to filter and re-encode data in a columnar format, using encoding algorithms like Run Length Encoding (RLE) and Bit Packing, while leveraging near-data processing capabilities to reduce data transmission and processing loads on the host processor.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is transmitted from SSD to host processor using PCIe connection, then data transfer can be performed, but the bandwidth is limited to about 4 GB per second which causes substantial transmission time
Solution Approach 1:
The patent extracts and removes unnecessary data from the transmission stream by applying filter predicates within the SSD before data leaves the device. This prevents unwanted data from being transmitted over the PCIe connection, effectively reducing the data volume and transmission time without requiring faster physical connections.
Solution Approach 2:
The patent performs data filtering and predicate evaluation actions within the SSD before data transmission occurs. By pre-processing the data and identifying which rows meet the query criteria while still stored in the columnar format, the system prepares the data in advance, reducing the amount of data that needs to be transmitted and processed by the host processor.
2Quantity of substance
If compressed or encoded data is transmitted to host processor, then storage efficiency is improved, but processing time increases due to decompression requirements
Solution Approach 1:
The patent enables the SSD to perform self-service by executing filter predicates and evaluating query conditions directly within the storage device using the in-storage compute engine. This eliminates the need to transfer compressed/encoded data to the host processor for filtering operations, allowing the SSD to serve itself by preparing filtered results before transmission.
Solution Approach 2:
The patent applies different processing approaches to different portions of data based on local requirements. Columnar data that meets predicate conditions is transmitted in its efficient columnar format, while data that doesn't meet conditions is filtered out locally. This allows selective optimization where only necessary data undergoes transmission and decompression.
3Ease of operation
If all columnar data is transmitted to host processor, then complete data availability is achieved, but bandwidth is wasted transmitting data that may not be needed
Solution Approach 1:
The patent performs preliminary filtering actions within the SSD by evaluating predicates against columnar data before transmission. This pre-processing step identifies and isolates only the relevant data rows that match query conditions, preparing a filtered subset for transmission that maintains data accessibility for the specific query while avoiding unnecessary bandwidth consumption.
Solution Approach 2:
The patent extracts only the necessary data rows from the columnar storage that satisfy the query predicates, removing unnecessary data from the transmission stream. This selective extraction ensures that the host processor receives only the data it needs for the specific query operation, optimizing bandwidth usage while maintaining ease of operation for the requested data access pattern.
Data Source
AI summary
A storage device is disclosed. The storage device may comprise storage for input encoded data. A controller may process read requests and write requests from a host computer on the data in the storage. An in-storage compute controller may receive a predicate from the host computer to be applied to the input encoded data. A transcoder may include an index mapper to map an input dictionary to an output dictionary, with one entry in the input dictionary mapped to an entry in the output dictionary, and another entry in the input dictionary mapped to a “don't care” entry in the output dictionary.