LSM Tree Block Property Indexing for Multi-Attribute Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional LSM tree storage systems lack efficient indexing for individual attributes in multi-attribute keys, leading to computationally intensive and inefficient queries, especially when searching ranges where the queried attribute is not a prefix of the key, and coarse-grained filtering at the sstable level introduces inefficiencies and correctness issues.
Innovation Solution
Implementing block properties in LSM tree storage systems by encoding and filtering data structures, including key-value entries with attributes, timestamps, and values, and using these properties to efficiently filter and locate relevant data blocks based on query criteria.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If multi-attribute keys are serialized into a single string key in LSM tree storage systems, then storage efficiency and simplicity are improved, but query efficiency deteriorates when searching for attributes that are not prefixes of the key
Solution Approach 1:
The patent segments the string key back into its constituent attributes by using the block property index to locate and parse specific attribute portions. This allows efficient querying of individual attributes without requiring the attribute to be a prefix, resolving the contradiction between simple serialized keys and efficient attribute-based queries.
Solution Approach 2:
The block property index acts as an intermediary structure that maps attribute names to their corresponding positions in the serialized key. This intermediary enables efficient attribute-based queries by translating query attributes into location information within the serialized key, eliminating the need for full key scans.
2Speed
If coarse-grained filtering is applied at the sstable level, then filtering speed is improved, but correctness and precision of search results deteriorate
Solution Approach 1:
The patent segments filtering from the sstable level down to the block level, and further to individual key-value entries. By using block properties that summarize specific attribute ranges at the block level, the system achieves both speed (by filtering at block level rather than scanning all entries) and precision (by using accurate attribute range summaries that preserve correctness).
Solution Approach 2:
The patent changes the filtering parameter from coarse-grained sstable-level filtering to fine-grained block-level filtering using block properties. This parameter change enables precise attribute range filtering while maintaining performance, as block properties provide accurate summaries of attribute ranges without requiring full sstable scans.
3Productivity
If block properties are added to enable efficient attribute-based queries, then query efficiency is improved, but device complexity and storage requirements increase
Solution Approach 1:
The patent merges the block property index with the existing LSM tree block structure, integrating attribute indexing capabilities into the standard block format. This merging approach enables efficient queries without adding separate complex indexing structures, as the block properties are encoded within the existing block handle and property arrays.
Solution Approach 2:
The block property structure serves multiple functions: it provides attribute-based filtering, enables efficient range queries, and maintains compatibility with existing LSM tree operations. This multi-functionality reduces overall system complexity by consolidating multiple indexing needs into a single unified structure.
Data Source
AI summary
Systems and methods for encoding block properties for data structures and filtering data structures using block properties are provided. A group of a plurality of key-value entries can be added to a data block of a data file. A block property indicative of attributes of the group of the plurality of key-value entries can be determined. The block property can be encoded in a block handle and added to an index block of the data file. A query directed to a plurality of data files can be received from a client device. A target block property included in the query can be compared to a block property corresponding to a data block of a data file of the plurality of data files. Based on the target block property intersecting with the block property of the data block, values corresponding to the data block can be sent to the client device.