Adaptive Database Clustering via ML Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems face inefficiencies due to the static nature of data clustering algorithms, leading to increased computational times and redundancy, as they fail to adapt dynamically to changing data distributions and query patterns.
Innovation Solution
A method and system for generating and using a classification model that dynamically selects the most efficient data clustering algorithm and table for queries based on performance information, employing machine learning models like recurrent neural networks to predict optimal clustering strategies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data clustering is performed to improve query efficiency, then query execution speed is improved, but computational overhead and processing time increase due to selecting the right clustering algorithm
Solution Approach 1:
The system pre-generates multiple candidate database tables with different clustering algorithms during an offline process. When a query arrives, the classification model quickly selects from these pre-prepared tables rather than performing clustering algorithm selection in real-time, thus improving query execution speed without significant computational overhead.
Solution Approach 2:
A classification model serves as an intermediary between the query and the database tables. The model takes query information as input and outputs the optimal pre-generated table, eliminating the need for runtime clustering algorithm selection and reducing computational time while maintaining query execution efficiency.
2Stability of the object's composition
If static data clustering algorithms are used to organize database tables, then data organization is achieved, but the system cannot adapt to changing data distributions and query patterns
Solution Approach 1:
The system dynamically adapts to changing data distributions and query patterns by using a trained classification model that learns from historical query performance data. The model can identify and select the most appropriate pre-generated tables based on current query characteristics, enabling the system to adapt without manual intervention while maintaining stable data organization through pre-computed tables.
Solution Approach 2:
The system incorporates feedback mechanisms where query performance information is collected and used to retrain the classification model. This feedback loop allows the system to learn from past performance and improve its ability to select optimal tables for changing data patterns, enhancing adaptability while preserving the stability of the underlying data organization.
3Adaptability or versatility
If multiple data clustering algorithms are applied to generate candidate tables, then query optimization options increase, but redundancy and unnecessary computation increase
Solution Approach 1:
The system extracts and separates the complex task of clustering algorithm selection from the query execution path. Multiple clustering algorithms are applied during an offline preprocessing stage to generate candidate tables, which are then stored for rapid selection during online query processing. This extraction reduces online system complexity while maintaining multiple optimization options through the pre-generated candidate tables.
Data Source
AI summary
The present disclosure provides a method for generating a classification model in a database system. The method comprises generating a plurality of database tables by performing data clustering on a database, acquiring a plurality of queries associated with information from the database, generating training data by executing each of the plurality of queries on the plurality of database tables, and training a classification model using the training data, wherein the classification model is used to select a database table for a query based on corresponding performance information associated with the query on the database during executing of the query. The present disclosure also provides a method for executing a query on a database using a classification model. The method comprises obtaining the classification model, obtaining a database query, generating a target table by inputting the database query into the classification model, and executing the database query on the target table. In addition, the present disclosure provides database systems and non-transitory computer-readable media for generating a classification model or executing a query on a database using a classification model.