Multi-Granularity Aggregation for Database Query Speed and Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In scenarios like finance and e-commerce, real-time data processing in databases faces challenges due to large-scale and continuous data arrival, requiring efficient aggregation methods to ensure real-timeness and accuracy of query results, especially in applications such as fraud surveillance and risk control where millisecond-level delays are critical.
Innovation Solution
The method involves performing aggregation calculations based on multiple distinct intervals, creating data clusters with different attribute value spans, and updating these clusters upon new data arrival, allowing for efficient querying of aggregate values by matching data clusters across layers, reducing the computational load and improving query speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is aggregated using a single aggregation interval, then the query speed is improved, but the accuracy of the query range is reduced
Solution Approach 1:
The patent divides the single aggregation interval into multiple aggregation intervals (first aggregation interval, second aggregation interval, etc.), creating multiple aggregation layers. Each layer aggregates data at different granularity levels, allowing the system to select appropriate layers based on query requirements to balance speed and accuracy.
Solution Approach 2:
The patent introduces a new dimension by creating multiple aggregation layers with different aggregation intervals rather than using a single interval. This multi-layer structure allows queries to traverse from coarser to finer granularity levels, enabling both fast approximate queries and accurate precise queries.
2Measurement precision
If multiple aggregation intervals are used to improve query range accuracy, then the aggregation calculation complexity increases
Solution Approach 1:
The patent performs preliminary aggregation calculations by pre-computing and storing aggregate values at multiple aggregation intervals in different aggregation layers. This preliminary action eliminates the need for complex real-time calculations during queries, as the system can directly retrieve pre-computed results or combine a small number of pre-computed results.
Solution Approach 2:
The patent merges the results from multiple aggregation layers to achieve the final query result. Instead of performing complex aggregation on raw data, the system combines pre-computed aggregate values from different layers, significantly reducing calculation complexity while maintaining accuracy.
3Use of energy by moving object
If data is processed in batch mode, then the computational resources are saved, but the real-timeness of data processing is reduced
Solution Approach 1:
The patent performs preliminary aggregation of data into multiple aggregation layers in advance, storing the aggregate values in the database. When queries arrive, the system can directly retrieve and combine these pre-computed results without performing full aggregation calculations, achieving both efficiency and real-timeness.
Solution Approach 2:
The patent implements a dynamic query mechanism that can adaptively select which aggregation layers to query based on the specific query requirements. For time-critical queries, the system can return results from coarser layers faster, while for accuracy-critical queries, it can combine results from multiple layers, dynamically balancing speed and precision.
Data Source
AI summary
Embodiments of this application provide a data processing method in a database. A plurality of aggregation intervals are selected, wherein the plurality of aggregation intervals are different from each other. And at least one piece of data in the database is aggregated separately based on each aggregation interval, so that data clusters stored in the database have data ranges with different attribute value spans.