Global AMQ Filter for LSM Database Secondary Index Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for updating secondary indexes in log-structured merge-tree (LSM) databases are inefficient due to high false positive rates in Bloom Filters, leading to unnecessary read-before-write and delete operations, especially in large databases.
Innovation Solution
Implementing a Global Approximate Member Query (AMQ) Filter, which is a probabilistic filter that supports deletions and resizing without hashing, to determine whether a primary key exists in the LSM database, thereby reducing the need for unnecessary operations by querying a single filter instead of multiple Bloom Filters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If Bloom Filters are used to check for primary key existence in LSM database, then the false positive rate increases in large databases, but the time and operations required for index updates increase due to unnecessary read-before-write and delete operations
Solution Approach 1:
The patent combines multiple Bloom Filters (one for each data file) into a single Global Bloom Filter that tracks all primary keys across the entire LSM database. This merging eliminates the need to query multiple separate filters, reducing the false positive rate while improving indexing performance by avoiding unnecessary read-before-write and delete operations.
Solution Approach 2:
The patent performs a preliminary check using the Global Bloom Filter before executing read-before-write or delete operations on the secondary index. By pre-determining whether a primary key already exists in the database through the filter query, the system avoids performing unnecessary operations, thus improving productivity while maintaining reliability.
2Measurement precision
If multiple Bloom Filters are queried to determine primary key existence, then the accuracy of existence determination improves, but the time and computational overhead increase
Solution Approach 1:
The patent merges multiple Bloom Filters into a single Global Bloom Filter that provides accurate primary key existence determination across the entire database. This single filter replaces the need to query multiple separate filters, maintaining measurement precision while significantly reducing the time and computational overhead associated with querying multiple filters during index updates.
Solution Approach 2:
The system performs a preliminary existence check using the Global Bloom Filter before proceeding with index update operations. This preliminary action accurately determines whether a primary key exists while minimizing time loss by avoiding the need to sequentially query multiple separate filters.
3Stability of the object's composition
If read-before-write and delete operations are performed on the secondary index for every update, then the index remains consistent with the database, but the operational efficiency decreases due to unnecessary operations
Solution Approach 1:
The patent performs a preliminary query of the Global Bloom Filter to determine whether a primary key already exists in the database before executing read-before-write or delete operations on the secondary index. This preliminary action ensures index consistency by performing operations only when necessary, while dramatically improving operational efficiency by eliminating unnecessary operations that would otherwise be performed for every update.
Data Source
AI summary
Systems and methods are described herein for efficiently updating a secondary index associated with a log-structured merge-tree (LSM) database. A Global approximate member query (AMQ) Filter is queried to determine whether a primary key, retrieved from a list of LSM database updates, already exists in the LSM database. If the primary key does not already exist in the LSM database then read-before-write and delete operations, typically performed with known approaches, do not need to be performed on the secondary index in order to update the secondary index, thereby avoiding significant additional computer processing and input/output operations.