Query-Oriented Approximate Query Processing Using Synthetic Data Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning-based approximate query processing models are insufficient for handling search queries with multiple conditions, leading to a significant decrease in accuracy and rendering them unusable for servicing such queries.
Innovation Solution
A query-oriented approximate query processing system is developed, which trains a sample generation model using a prior model, proposal model, and generative model to generate synthetic data entries that account for conditions in search queries, optimizing an objective function to reduce reconstruction error and statistical distance, and deploying this model to client devices for query processing without accessing the original dataset.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing ML-model-based AQP techniques are used to process search queries, then the system can provide approximate results efficiently, but the accuracy of results significantly decreases when search queries include multiple conditions
Solution Approach 1:
The system performs preliminary training of the sample generation model using query workload information and dataset characteristics before actual query processing. This preliminary action enables the model to learn the data distribution and query patterns in advance, ensuring high accuracy when multiple conditions are applied during runtime without sacrificing processing efficiency
Solution Approach 2:
The system creates synthetic data copies through the trained sample generation model that replicate the statistical properties and conditional relationships of the original dataset. These synthetic copies can be queried efficiently with multiple conditions while maintaining accuracy, as they are generated to preserve the underlying data distribution and attribute correlations
2Device complexity
If AQP models are trained without accounting for search queries, then the model structure remains simple, but the model becomes unusable for servicing search queries with multiple conditions
Solution Approach 1:
The sample generation model is designed with multi-functionality by incorporating query workload information during training. The model can handle various types of search queries with different numbers of conditions while maintaining a unified architecture. This universal design allows the same model structure to adapt to diverse query patterns without requiring separate specialized models for each query type
3Loss of energy
If the trained sample generation model is deployed to client computing devices, then network bandwidth and computational resource consumption are reduced, but the model size and initial storage requirements increase
Solution Approach 1:
The system extracts only the essential sample generation model from the training environment and deploys it to client devices, leaving behind the large original dataset and training infrastructure. This extraction enables clients to generate synthetic data locally without needing access to the original data or training resources, significantly reducing network bandwidth and computational resource consumption during query processing
Data Source
AI summary
In some embodiments, a model training system trains a sample generation model configured to generate synthetic data entries for a dataset. The sample generation model includes a prior model for generating an estimated latent vector from a partially observed data entry, a proposal model for generating a latent vector from a data entry of the dataset and a mask corresponding to the partially observed data entry, and a generative model for generating the synthetic data entries from the latent vector and the partially observed data entry. The model training system trains the sample generation model to optimize an objective function that includes a first term determined using the synthetic data entries and a second term determined using the estimated latent vector and the latent vector. The trained sample generation model can be executed on a client computing device to service queries using the generated synthetic data entries.