Bloom Filter Partition Checks Before Database Slice Loading
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The process of loading data from disk to memory is time-consuming and resource-intensive, often resulting in unnecessary operations when the requested data is not included in the loaded portion, necessitating efficient optimization.
Innovation Solution
Implementing a used identifier cache and bloom filters to determine whether requested data is present in a data slice before loading, using a bloom filter management system that creates filters based on frequently used identifiers to minimize resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is loaded from disk to memory without pre-checking, then data retrieval can proceed immediately, but unnecessary loading operations consume time and resources
Solution Approach 1:
The patent applies preliminary action by creating bloom filters before data loading operations. The system pre-processes data to generate bloom filter structures that can quickly determine whether requested data exists in a partition. This allows the system to check data presence before initiating loading operations, avoiding unnecessary disk-to-memory transfers and reducing overall retrieval time.
2Reliability
If bloom filters are created for all identifiers, then data presence can be accurately checked, but memory resources are excessively consumed
Solution Approach 1:
The patent applies local quality by creating bloom filters selectively rather than uniformly for all identifiers. The system identifies frequently queried identifiers and creates bloom filters only for those, while using different strategies for less frequently accessed data. This localized approach maintains detection accuracy for common queries while significantly reducing overall memory consumption.
Solution Approach 2:
The patent applies parameter changes by adjusting bloom filter characteristics such as filter size, hash function count, and false positive rate based on identifier frequency and data access patterns. This allows optimization of the balance between detection reliability and memory usage, allocating more resources to high-priority identifiers and fewer resources to low-priority ones.
Data Source
AI summary
Embodiments are described for a database management system comprising a memory and at least one processor coupled to the memory. The at least one processor is configured to receive a query that corresponds to a data slice and determine a bloom filter based on the query. The at least one processor is further configured to determine that the data slice includes data requested by the query based on the bloom filter and in response to determining that the data slice includes the data requested by the query, load the data slice to the memory.