Neural Network Training Set Generation for Database Query Approximation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing solutions for retrieving insights from large and unknown datasets in databases are inefficient, as they often rely on indexing or caching algorithms that may not be programmed to process such data, leading to prolonged query times.
Innovation Solution
A method is introduced to generate training sets for neural networks by determining segmentation based on a column from a columnar database table, generating group-by queries, and creating reduced queries to produce training query pairs, which can be used to approximate query results more efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional database querying methods are used on large datasets, then query accuracy is maintained, but query execution time becomes excessively long
Solution Approach 1:
The system performs preliminary actions by generating training sets from historical query data and pre-training neural network models before actual queries are executed. This allows the system to have approximation capabilities ready in advance, enabling fast initial query results without executing full database scans each time.
Solution Approach 2:
A neural network model serves as an intermediary between the user's query and the actual database execution. The model first provides approximate results rapidly, and only when higher precision is needed does the system execute the full query on the database, thus mediating between speed and accuracy requirements.
2Speed
If indexing and caching algorithms are used to accelerate database access, then query speed is improved for known datasets, but effectiveness decreases for large and unknown datasets
Solution Approach 1:
The neural network model continuously learns and adapts from actual query patterns and results, improving its own performance over time without external reprogramming. The system automatically updates its understanding of data distributions and query behaviors, making it increasingly effective for both known and unknown datasets.
Solution Approach 2:
The system changes parameters by transforming structured database queries into neural network input formats, adjusting model architecture and training parameters based on data characteristics. This allows the model to adapt to different data types, sizes, and query patterns dynamically.
3Measurement precision
If full database queries are executed to ensure accurate results, then measurement precision is maintained, but productivity decreases due to long processing times
Solution Approach 1:
The system performs partial action by executing only necessary portions of queries through the neural network approximation first. Full database queries are executed only when the approximation confidence is below a threshold or when exact precision is required, avoiding unnecessary complete scans for queries that can be adequately approximated.
Data Source
AI summary
A system and method for generating training sets for training neural networks. The method includes determining a segmentation based on a column from a columnar database table; generating a group-by query based on the segmentation; generating a plurality of reduced queries based on the group-by query; executing the group-by query on a table of a database to obtain a result table, wherein the result table includes a plurality of results, wherein each result corresponds to a respective reduced query of the plurality of reduced queries; and generating a plurality of training query pairs by pairing each reduced query with its corresponding reduced result.