Query Clustering for Anomalous Database Query Identification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern relational database systems face challenges in optimizing and securing a large volume of queries due to their inability to efficiently process and classify queries, leading to resource inefficiencies and potential security risks from anomalous or malicious queries.
Innovation Solution
The system encodes and classifies queries using clustering techniques, such as k-means clustering, to identify anomalous or suspicious queries by generating query vectors and grouping similar queries, allowing for the detection and prevention of potentially harmful queries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a query optimizer evaluates different query execution plans for each query, then query execution efficiency is improved, but the system becomes overwhelmed by the inordinate number of queries and cannot optimize them all
Solution Approach 1:
The patent segments queries into clusters based on similarity metrics (e.g., query structure, tables accessed, operations performed). Queries within the same cluster share common optimization characteristics, allowing the system to evaluate one execution plan per cluster rather than per query. This segmentation reduces the optimization burden from millions of individual queries to a manageable number of clusters while maintaining execution efficiency for each query type.
Solution Approach 2:
The system performs preliminary clustering and execution plan evaluation before actual query execution. By pre-grouping similar queries and determining optimal execution plans for each cluster in advance, the system avoids repeated optimization work for similar queries. When a new query arrives, it is quickly matched to an existing cluster and executed with the pre-determined plan, eliminating real-time optimization overhead.
2Productivity
If the database system executes all received queries, then complete query processing is achieved, but resource wastage occurs due to malicious or inefficient queries
Solution Approach 1:
The system performs preliminary analysis during query clustering to identify and flag potentially harmful or inefficient queries before execution. By examining query patterns, complexity metrics, and cluster characteristics in advance, the system can pre-determine which queries should be executed, modified, or rejected. This preliminary filtering prevents resource wastage on malicious or inefficient queries while ensuring legitimate queries are processed completely.
Solution Approach 2:
The system incorporates feedback mechanisms that analyze query execution results and cluster performance metrics to continuously improve query filtering and optimization. Execution statistics, error patterns, and resource consumption data feed back into the clustering and plan selection processes, enabling the system to learn from past queries and better identify harmful patterns. This feedback loop ensures complete processing of valid queries while progressively improving rejection accuracy for harmful ones.
3Productivity
If the system performs comprehensive query optimization, then execution performance is improved, but the time required to process each query increases
Solution Approach 1:
By segmenting queries into clusters with similar characteristics, the system performs optimization work once per cluster rather than once per query. The clustering process groups queries that share execution patterns, allowing the expensive optimization step to be amortized across multiple queries. This segmentation dramatically reduces total optimization time while maintaining high execution performance for each query type.
Solution Approach 2:
The system performs execution plan evaluation and optimization in advance during the clustering phase, before actual query execution. By pre-determining optimal execution plans for each cluster based on historical data and query patterns, the system eliminates real-time optimization delays. When queries arrive, they are quickly matched to pre-optimized clusters and executed immediately with the predetermined plans, minimizing processing time.
4Quantity of substance
If the database system stores and processes all queries, then data completeness is maintained, but storage needs increase due to redundant query data
Solution Approach 1:
The patent merges similar queries into clustered groups based on structural and semantic similarities. Instead of storing and processing each query independently, the system consolidates redundant queries into representative cluster members. This merging reduces the total volume of stored query data while preserving the essential information needed for execution and optimization, as all queries in a cluster share common execution plans and characteristics.
Solution Approach 2:
The system segments the query data space into distinct clusters, storing only the essential representative queries and cluster metadata rather than all individual queries. Each cluster is stored as a unit with its own execution plan, eliminating the need to redundantly store identical or near-identical query definitions. This segmentation approach maintains data completeness for execution purposes while dramatically reducing storage requirements.
Data Source
AI summary
Techniques are disclosed relating to managing database queries. In some embodiments, a server system receives a query from a computer system and determines a set of aspects for the query, including at least a number of columns specified in the query and a computational cost of executing the query. The system generates a query vector based on the set of aspects determined for the query. The system then compares the query vector with a plurality of clusters, ones of the plurality of clusters comprising two or more previously generates query vectors generated based on aspects of queries previously received by the server system. Based on the comparing, specifically a distance between the query vector and the plurality of clusters of previously generated query vectors, the system classifies the query. Based on a classification of the query determined during the classifying, the system manages the query.