Big Data Stress Testing via Query Distance Ranking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Stress testing of big data management systems faces challenges in generating optimal workloads that effectively test reliability, resiliency, and stability within limited time and resource constraints, especially with the transition to big data clusters, where the number of executable complex queries within a 24-hour period significantly decreases due to the large volume of data.
Innovation Solution
A computer-implemented method that measures the distance between data points of random test queries, ranks queries exceeding a threshold distance, and executes them to identify an optimal set of queries for stress testing, ensuring efficient use of time and resources without compromising quality, by utilizing a system that generates and prioritizes queries based on access plan characteristics and target table metadata.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a large number of complex queries are executed for stress testing, then test coverage and quality improve, but the time required and computational resources increase significantly
Solution Approach 1:
The system performs preliminary analysis of query characteristics and access plan features before executing stress tests. By pre-evaluating queries and identifying those with high potential to precipitate errors based on their distance from normal query patterns, the system prepares an optimized test subset in advance, reducing the time needed during actual stress testing while maintaining comprehensive coverage
Solution Approach 2:
The system extracts and selects only the most critical queries from a large pool of potential test queries. By measuring the distance between query data points and identifying outliers that exceed a threshold, the system extracts a smaller subset of high-value queries that are most likely to reveal system failures, thereby reducing the total number of queries needed while maintaining test quality
2Reliability
If more queries are executed to ensure comprehensive stress testing, then test coverage improves, but resource consumption increases
Solution Approach 1:
The system extracts a representative subset of queries that maximize test coverage while minimizing resource usage. By analyzing query characteristics and selecting only those with high error-induction potential based on distance metrics, the system reduces the total computational burden while maintaining comprehensive stress testing coverage
Solution Approach 2:
The system changes the parameters of query selection by introducing distance-based filtering and thresholding. Instead of executing all queries or using random selection, the system transforms the selection process by measuring distances in feature space and selecting queries that exceed threshold values, thereby optimizing the balance between coverage and resource consumption
3Ease of operation
If random queries are selected for stress testing, then simplicity of selection is maintained, but effectiveness in precipitating errors decreases
Solution Approach 1:
The system performs preliminary analysis of query characteristics and establishes distance metrics before selection. By pre-computing features of queries and determining their distances from normal patterns, the system maintains an automated, straightforward selection process while significantly improving error precipitation capability through data-driven rather than purely random selection
Solution Approach 2:
The system introduces distance metrics and threshold values as intermediaries between random query generation and actual query selection. These intermediaries filter and prioritize queries based on their likelihood to precipitate errors, maintaining the simplicity of automated selection while dramatically improving effectiveness through objective, measurable criteria
Data Source
AI summary
A computer-implemented method, system and computer program product for optimally performing stress testing against big data management systems. A set of random test queries is generated and compiled to determine the data points of the features (e.g., table type being queried) of the set of random test queries. A distance (e.g., Mahalanobis distance) is then measured between the data points of the features and the mean of a distribution of data points corresponding to each same feature of an extracted feature set. Each random test query whose distance exceeds a threshold distance is then ranked. The ranked random test queries are then executed in order of rank. Those executed random test queries which resulted in an error (e.g., system failure) are added to a log, which is used to identify those queries to perform a stress test against the big data management system.