Query Splitting for Log Data Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern distributed information systems face challenges in efficiently retrieving and analyzing large volumes of log data due to limited query capabilities of database platforms, necessitating more customizable and efficient techniques for searching and retrieving log data.
Innovation Solution
The technique involves splitting a user query into native query operations and post-processing operations, where native query operations are executed by the database platform, and post-processing operations are executed on the results by the management plane, allowing for more flexible and efficient data retrieval and analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If all query operations are executed by the database platform, then query execution is simplified, but query functionality is limited and cannot handle computationally expensive operations
Solution Approach 1:
The query is divided into two distinct parts: native query operations that are executed by the database platform and post-processing operations that are executed by the management plane. This segmentation allows the system to leverage the database platform's efficient data retrieval capabilities while also supporting complex computational operations that require custom logic, thereby resolving the contradiction between query functionality and execution complexity.
2Measurement precision
If computationally expensive operations are executed by the database platform, then query results are more accurate, but execution cost and time increase
Solution Approach 1:
Computationally expensive post-processing operations are extracted from the database platform and executed separately by the management plane. This extraction allows the database platform to focus on efficient data retrieval while the management plane handles complex computations using cached data, thereby maintaining query result accuracy while reducing execution cost and time.
3Adaptability or versatility
If the database platform supports more query operations, then query flexibility increases, but platform complexity and resource consumption increase
Solution Approach 1:
The management plane acts as an intermediary between the user and the database platform. It receives the full query from the user, separates native operations (executed by the database platform) from post-processing operations (executed by the management plane itself), and coordinates their execution. This intermediary approach enables query flexibility without increasing database platform complexity, as the management plane handles the complex operations using cached data.
Data Source
AI summary
Techniques for splitting a query into native query operations and post-processing operations are provided. In one embodiment, a user query including a plurality of search parameters is received. Expressions representing the plurality of search parameters in the user query are determined. A first subset and a second subset of the expressions are identified. Each expression of the first subset corresponds to one of a plurality of native query operations executable by a database platform, and each expression of the second subset corresponds to one of a plurality of predefined post-processing operations not executable by the database platform. A database query is generated based on one or more expressions of the first subset, and a post-processing query is generated based on each expression in the second subset. The database query is executed by the database platform, then the post-processing query is executed on the results from database platform.