Hybrid Storage With Offline Row-to-Column Conversion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computational storage devices face inefficiencies in data processing and storage, particularly in handling row-based data formats, leading to high latency, low throughput, and high power consumption during inline conversions.
Innovation Solution
A hybrid storage technique that converts row-based data into column-based format offline, using computational storage devices with embedded processors, allowing efficient data processing and storage, reducing latency and power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If row-based data format is used in computational storage devices, then data can be stored in traditional format, but latency is high and throughput is low during inline conversions
Solution Approach 1:
The patent converts row-based data to column-based format in advance during offline processing, before the data is stored in the computational storage device. This preliminary conversion eliminates the need for time-consuming inline conversions during query execution, thereby reducing latency and improving throughput when the device retrieves and processes data.
2Ease of operation
If inline row-to-column conversion is performed, then data can be processed in column-based format, but power consumption is high
Solution Approach 1:
The data conversion from row-based to column-based format is performed offline in advance, before the data is loaded into the computational storage device. This eliminates the need for power-consuming inline conversion operations during device operation, significantly reducing power consumption while maintaining the ability to process data efficiently in column-based format.
3Quantity of substance
If column-based format is used, then storage space is optimized and computational efficiency is improved, but data must be converted from row-based format
Solution Approach 1:
The conversion from row-based to column-based format is performed offline during the data preparation phase, before the data is transferred to the computational storage device. This approach achieves the storage space efficiency and computational benefits of column-based format without adding complexity to the device itself, as the conversion is handled by external processing systems.
Data Source
AI summary
Embodiments of the present disclosure are directed to a method for storing and processing data. The method includes identifying a database in a memory of a host device having one or more rows and one or more columns. A partition having a partition size is identified, and the one or more rows of the database is identified based on the partition size. The data stored in the one or more rows is converted into a column-based format, and the data is stored in a computational storage device in the column-based format. The computational storage device is configured to retrieve the data stored in the column-based format, in response to a query, and process the query based on the data.