LSM-Tree Data Reading via Semi Join Predicate Pushdown
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
LSM-Tree databases face challenges in pushing down predicates in the Value column to the storage layer during a Merge-On-Read operation, leading to incorrect query results.
Innovation Solution
A data reading method that involves receiving a query condition for the Value column, querying first alternative data, constructing a right table based on the Keys of the first alternative data, performing a Semi Join with the total database data, and finally querying target data from the second alternative data to achieve predicate pushdown.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If predicate pushdown is directly applied to the Value column in Merge-On-Read, then query performance improves, but query correctness deteriorates
Solution Approach 1:
The query processing is segmented into three distinct phases: (1) preliminary filtering of candidate data based on Value column predicates, (2) Semi Join operation to identify latest versions by matching Keys between candidate data and total data, and (3) final filtering to obtain target data. This segmentation allows each phase to focus on a specific aspect, ensuring both performance and correctness.
Solution Approach 2:
The method performs preliminary filtering of the Value column before the final query execution. By first identifying candidate data that satisfies the Value condition and then using Semi Join to determine which candidates represent the latest versions, the system prepares the data in advance for the final correct result extraction, improving overall query efficiency.
2Reliability
If all data is scanned to ensure correctness in Merge-On-Read, then query correctness improves, but data scanning volume increases
Solution Approach 1:
The method extracts only the necessary information for correctness verification through the Semi Join operation. Instead of scanning all data, it extracts the Key information from candidate data and performs a targeted join with total data to identify latest versions, significantly reducing the scanning volume while maintaining correctness.
Solution Approach 2:
The method performs a partial scan by first filtering candidate data based on Value predicates, then only scanning the necessary portions of total data through Semi Join to verify latest versions. This partial action approach avoids excessive full-data scanning while ensuring correctness through targeted verification.
Data Source
AI summary
The present disclosure provides a data reading method and device for a database and storage medium, which includes: receiving a query condition for a Value column in an LSM-Tree database; querying first alternative data with Value meeting the query condition from the LSM-Tree database; constructing a right table according to Key of the first alternative data, taking total data in the LSM-Tree database as a left table, and carrying out Semi Join based on the left table and the right table, so as to acquire second alternative data matched with each Key of the right table from the left table; querying target data with Value meeting the query condition from the second alternative data. The present disclosure can realize a predicate pushdown function of an LSM-Tree database in Merge-On-Read scene, in which, a large amount of data scanning is avoided while write performance and query performance are balanced.