Conditional Column Transcoding With Predicate Filtering in SSDs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current storage devices, such as Solid State Drives (SSDs), face challenges in efficiently transferring data to host processors due to bandwidth limitations and the need to process compressed or encoded data, which can result in increased data transmission and processing times.
Innovation Solution
Implementing a transcoder within the storage device to filter and re-encode data in a columnar format, using encoding algorithms like Run Length Encoding (RLE) and Bit Packing, while leveraging near-data processing capabilities to reduce data transmission and processing loads on the host processor.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is transmitted from SSD to host processor using PCIe connection, then data transfer capability is provided, but bandwidth limitation results in slow data transfer speed
Solution Approach 1:
The patent extracts and removes unnecessary data from the transmission stream by applying filter predicates within the SSD before data leaves the device. This prevents unwanted data from being transmitted over the PCIe connection, directly reducing the quantity of transmitted data and improving effective transfer speed for the required information.
Solution Approach 2:
The patent performs data filtering and predicate evaluation actions preliminarily within the SSD before data transmission occurs. By pre-processing the data and identifying which records satisfy the query conditions, the system avoids transmitting unnecessary data, thereby improving the effective data transfer speed without being constrained by PCIe bandwidth limitations.
2Quantity of substance
If compressed or encoded data is transmitted to host processor, then storage efficiency is improved, but processing time increases due to decompression/decoding requirements
Solution Approach 1:
The patent implements self-service by performing predicate evaluation and filtering operations directly on the compressed or encoded data within the SSD, without requiring full decompression or decoding first. The filter operates on the encoded representation, and only data satisfying the predicates is transmitted, thus avoiding the time-consuming decompression step for unnecessary data.
Solution Approach 2:
The system performs preliminary filtering on encoded data using bitwise operations and predicate evaluation before transmission. This preliminary action identifies and isolates only the necessary encoded records, preventing the host processor from wasting time decompressing or decoding data that will ultimately be discarded anyway.
3Loss of information
If all data is transmitted to host processor for processing, then complete data availability is achieved, but host processor workload and processing time increase
Solution Approach 1:
The patent extracts only the specific data records that satisfy the query predicates from the full dataset stored in the SSD. By evaluating filter conditions within the storage device and transmitting only matching records, the system maintains data completeness for the required information while dramatically reducing the volume of data sent to the host processor, thereby improving processor efficiency.
Solution Approach 2:
The patent introduces an intermediary filtering layer within the SSD that acts as a mediator between the stored data and the host processor. This intermediary evaluates predicates and selectively transmits only relevant data, protecting the host processor from unnecessary workload while ensuring that complete and accurate data for the query is provided.
4Speed
If data is stored in columnar format, then query performance is improved, but data re-encoding and filtering complexity increases
Solution Approach 1:
The patent replaces complex mechanical or computational transcoding operations with efficient bitwise operations and predicate evaluations that can be performed directly on the encoded columnar data. Instead of fully decoding and re-encoding data, the system uses bitwise logic to filter columnar data in its encoded form, maintaining query performance while reducing transcoding complexity.
Data Source
AI summary
A storage device is disclosed. The storage device may comprise storage for input encoded data. A controller may process read requests and write requests from a host computer on the data in the storage. An in-storage compute controller may receive a predicate from the host computer to be applied to the input encoded data. A transcoder may include an index mapper to map an input dictionary to an output dictionary, with one entry in the input dictionary mapped to an entry in the output dictionary, and another entry in the input dictionary mapped to a “don't care” entry in the output dictionary.