Database Query Apparatus Using Deep Learning for Dynamic Partition Probing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face challenges in determining the optimal number of data partitions to query (probing cardinality) for efficient data retrieval, leading to either increased query latency or reduced result recall.
Innovation Solution
The proposed solution involves using a pre-trained deep learning model to predict the number of data partitions to query based on the query vector, result number, and distances between the query vector and the center vector of each data partition, thereby dynamically determining the probing cardinality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the probing cardinality is set as a larger value, then the number of data partitions to be queried increases and result recall improves, but query latency increases and efficiency decreases
Solution Approach 1:
The patent applies dynamics by transitioning from a fixed probing cardinality to a dynamic one. The system uses a deep learning model that takes the query vector and distance information as inputs to predict the optimal probing cardinality for each specific query request. This allows the number of data partitions to be queried to adapt dynamically based on the query characteristics, thereby improving result recall when needed while reducing query latency when fewer partitions suffice.
Solution Approach 2:
The patent changes the parameter of probing cardinality from a static constant to a dynamic value predicted by a deep learning model. The model processes the query vector and distance information to output an optimal probing cardinality value, effectively changing this critical parameter based on the specific query request. This parameter change enables the system to optimize the trade-off between result recall and query latency for each query.
2Productivity
If the probing cardinality is set as a smaller value, then query efficiency improves and query latency reduces, but result recall decreases
Solution Approach 1:
The system dynamically adjusts the probing cardinality based on the query characteristics by using a deep learning model. Instead of using a small fixed value that always limits result recall, the model predicts the appropriate probing cardinality for each query, allowing the system to maintain high query efficiency when fewer partitions are needed while ensuring adequate result recall when more partitions should be queried.
Solution Approach 2:
The probing cardinality parameter is transformed from a small fixed constant to a dynamically predicted value. The deep learning model processes query-specific information to determine the optimal probing cardinality, enabling the system to achieve high query efficiency without sacrificing result recall by appropriately adjusting this parameter based on the specific query request.
3Ease of operation
If a fixed probing cardinality is used for all query requests, then the system complexity is low and ease of operation is high, but the system cannot adapt to different query characteristics leading to suboptimal performance
Solution Approach 1:
The system applies self-service by using an automated deep learning model to determine the optimal probing cardinality for each query request. Instead of requiring manual configuration or complex rule-based logic, the model automatically processes the query vector and distance information to predict the appropriate probing cardinality. This maintains ease of operation while significantly improving query adaptability to different query characteristics.
Solution Approach 2:
The system implements feedback by using the distance information between the query vector and center vectors of data partitions as input to the deep learning model. This feedback mechanism allows the model to adapt the probing cardinality based on the specific spatial relationships in the query, improving query adaptability while the automated nature of the model preserves system simplicity.
Data Source
AI summary
A database system that, after receiving the query request, adopts a pre-trained deep learning model to predict probing cardinality corresponding to the query request based on the query vector in the query request, the number of vectors of the query result, and the vector corresponding to the distances between the query vector and the center vector of a plurality of data partitions stored in the storage device. The system then determines the target data partitions according to the probing cardinality, and obtains the query result from the target data partitions. Since the probing cardinality is dynamically determined based on the query request, decreases to both the query efficiency due to a larger setting of the probing cardinality, and the query recall due to a smaller setting of the probing cardinality are avoided, which is beneficial in reducing the average number of target data partitions and improving query efficiency.