Predicate-Object Name Cache for Database Query Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database management systems (DBMS) face inefficiencies when processing queries against large object stores in the cloud, as they retrieve and process unnecessary data, leading to high latency and uneven workloads across processing engines due to the retrieval of all objects before filtering by query predicates.
Innovation Solution
Implementing a predicate-object name cache in the DBMS memory to store mapping information between query predicates and relevant object names, allowing for the retrieval of only necessary objects and distributing them evenly across processing engines, thereby reducing unnecessary data retrieval and workload skew.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all objects are retrieved from remote data store before filtering by query predicates, then complete data is available for processing, but data retrieval time and network bandwidth consumption increase significantly
Solution Approach 1:
The system performs preliminary actions by evaluating query predicates against object metadata (names, prefixes, paths) before retrieving actual object data. This allows the system to pre-determine which objects are relevant to the query, filtering out irrelevant objects before they are retrieved from remote storage, thus reducing retrieval time while ensuring data completeness for relevant objects only
Solution Approach 2:
The system extracts and utilizes object metadata (names, prefixes, paths) separately from the actual object data. By taking out this metadata and using it for predicate evaluation, the system can identify and retrieve only the necessary objects from remote storage, rather than retrieving all objects and then filtering them, thereby reducing data retrieval time and network bandwidth consumption
2Reliability
If all objects are retrieved and processed by all processing engines, then query results are comprehensive, but workload becomes uneven across engines and processing efficiency decreases
Solution Approach 1:
The system segments the workload by distributing different sets of relevant objects to different processing engines based on predicate evaluation results. Each processing engine receives only the objects that are relevant to its assigned query predicates, rather than all objects being processed by all engines. This segmentation ensures comprehensive query results while balancing the workload across engines, thereby improving processing efficiency
Solution Approach 2:
The system applies local quality by tailoring the data distribution to each processing engine's specific query predicates. Each engine receives a customized set of objects that are locally relevant to its query requirements, rather than a uniform distribution of all objects. This localized approach ensures that each engine processes only necessary data, improving overall processing efficiency while maintaining query result completeness
3Use of energy by moving object
If object metadata is not cached, then memory usage is reduced, but repeated predicate evaluations against remote objects increase processing time
Solution Approach 1:
The system performs preliminary action by caching object metadata (names, prefixes, paths) in memory before query processing. This cached metadata is used for rapid predicate evaluation without requiring repeated accesses to remote storage. The preliminary caching of metadata improves query processing speed by enabling fast filtering, while the caching strategy is designed to balance memory usage by storing only essential metadata rather than complete object data
Data Source
AI summary
In some examples, a database system includes a memory to store a predicate-object name cache, where the predicate-object name cache contains predicates mapped to respective object names. The database system further includes at least one processor to receive a query containing a given predicate, identify, based on accessing the predicate-object name cache, one or more object names indicated by the predicate-object name cache as being relevant for the given predicate, retrieve one or more objects identified by the one or more object names from a remote data store, and process the query with respect to data records of the one or more objects retrieved from the remote data store.