LSM Tree Data Storage Engine Row to Columnar Format Conversion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing data storage methods using Log-Structured Merge (LSM) trees, such as RocksDB, face inefficiencies in range queries due to storing data in row format on disk, which hampers statistical analysis efficiency.
Innovation Solution
The proposed solution involves storing new data in row format in RAM and converting it to columnar format when the disk storage threshold is reached, allowing for efficient storage and retrieval of data in columnar format during range queries, thereby improving query efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If data is stored in row format on disk, then storage simplicity is maintained, but range query efficiency deteriorates
Solution Approach 1:
The patent divides the storage system into two distinct segments: a mutable table in RAM for new data ingestion and a disk storage layer for historical data. The disk storage further segments data by storing it in columnar format rather than row format, creating a specialized structure optimized for range queries and statistical operations on historical data while maintaining simple row-based storage for new data.
Solution Approach 2:
The patent applies different storage formats to different locations in the storage hierarchy. New data in the mutable table and lowest disk level uses row format for simplicity and fast ingestion, while historical data in upper disk levels uses columnar format for optimized range query performance. This local differentiation allows each storage location to have the quality (storage format) best suited for its specific access patterns.
2Productivity
If data is converted to columnar format, then range query efficiency is improved, but storage complexity increases
Solution Approach 1:
The patent performs data format conversion in advance during the compaction process. When data is moved from the mutable table to upper levels of disk storage, it is pre-converted to columnar format before being stored. This preliminary action ensures that when range queries are executed on historical data, the data is already in the optimal columnar format, eliminating the need for runtime format conversion and reducing query complexity.
Solution Approach 2:
The patent introduces a compaction process as an intermediary mechanism that handles the complex task of format conversion between row format and columnar format. This intermediary process manages the transformation automatically during data compaction, shielding users from the complexity of format conversion while enabling efficient columnar storage for historical data.
3Reliability
If mutable table is converted to immutable table, then data consistency is improved, but storage flexibility deteriorates
Solution Approach 1:
The patent segments the storage structure into mutable tables in RAM for active data ingestion and immutable tables on disk for stable historical storage. This segmentation allows the mutable portion to remain flexible and adaptable for new data, while the immutable portion provides consistency and reliability for historical data. Each segment serves its specific purpose without compromising the other.
Solution Approach 2:
Instead of making the entire storage system immutable for consistency, the patent inverts the approach by making only the historical disk storage immutable while keeping the mutable table in RAM flexible. This inversion allows the system to achieve data consistency for historical data without sacrificing the adaptability needed for continuous data ingestion and updates in the mutable portion.
Data Source
AI summary
A data storage method includes the following steps: storing data to be stored into a mutable table, the mutable table being located in a RAM. If a storage content of the mutable table is greater than or equal to a preset storage content threshold, modifying the mutable table into an immutable table and stopping receiving storage of the data to be stored; if a storage content of a lowest level in a disk is greater than or equal to a storage content threshold of a present level, and next level higher in the disk is stored with columnar format, storing data of the lowest level to the next level in the columnar format; and if the lowest level is stored with row format, storing data in the immutable table to the lowest level in the row format.