Conditional Transcoder Predicates for Column Store Bandwidth Limits
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage devices face challenges in efficiently transferring data to host processors due to bandwidth limitations and the inefficiencies of near-storage processing on compressed or encoded data, which can result in increased data transmission and processing times.
Innovation Solution
Implementing a transcoder within the storage device to process encoded data in a columnar format, utilizing encoding algorithms like Run Length Encoding (RLE) and Bit Packing, and leveraging an Index Mapper to filter and modify encoding without inflating the data, thereby reducing the amount of data sent to the host processor.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is transmitted from storage device to host processor using conventional bandwidth-limited interfaces, then data transfer can be performed, but the transfer speed is limited and takes substantial time
Solution Approach 1:
The patent extracts and processes only the necessary data columns at the storage device level using columnar storage format, rather than transferring all data to the host processor. This selective extraction of required data reduces the volume of transmitted data and accelerates transfer speed by avoiding unnecessary data movement across the bandwidth-limited interface.
Solution Approach 2:
The patent performs preliminary data organization and filtering operations at the storage device before data transfer. By pre-processing data into columnar format and identifying which columns are needed, the system prepares data in advance, reducing the amount of data that needs to be transferred and the time required for host-side processing.
2Productivity
If near-storage processing is performed on compressed or encoded data, then processing can be done closer to data, but it results in increased data transmission and processing times
Solution Approach 1:
The patent changes the data representation parameters by using columnar storage format and various encoding schemes (Run Length Encoding, Bit Packing, Dictionary Encoding). These parameter changes allow the data to be stored and processed in a more efficient format that reduces the computational overhead of near-storage processing while minimizing the time required for data transformation and transmission.
3Reliability
If all data is transmitted to host processor for processing, then complete data is available for operations, but bandwidth limitations cause substantial transmission time
Solution Approach 1:
The patent extracts only the specific columns required by the query from the columnar storage format, rather than transmitting all data. This selective extraction maintains data reliability for the specific processing needs while dramatically reducing the volume of data transmitted across the bandwidth-limited interface, thus reducing transmission time without sacrificing the completeness of required data.
Data Source
AI summary
A storage device is disclosed. The storage device may comprise storage for input encoded data. A controller may process read requests and write requests from a host computer on the data in the storage. An in-storage compute controller may receive a predicate from the host computer to be applied to the input encoded data. A transcoder may include an index mapper to map an input dictionary to an output dictionary, with one entry in the input dictionary mapped to an entry in the output dictionary, and another entry in the input dictionary mapped to a “don't care” entry in the output dictionary.