Doris History Aggregation Caching for Real-Time Query Concurrency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional caching mechanisms in Doris, such as SQL Cache and Partition Cache, have poor concurrency effects for real-time data queries, leading to inefficient use of IO and CPU resources due to the need for full recalculations and metadata overload.
Innovation Solution
Cache the aggregation results of history data in a preset system, merging them with real-time data to reduce redundant calculations and improve concurrency by leveraging a key-value pair storage structure and using a least recently used algorithm for cache management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If SQL Cache and Partition Cache are used to improve data query concurrency, then offline query performance is improved, but real-time data query concurrency remains poor
Solution Approach 1:
The patent segments the data into history data and real-time data, and segments the caching strategy accordingly. History data aggregation results are cached using SQL Cache and Partition Cache, while real-time data is processed separately through incremental aggregation, allowing each segment to be optimized for its specific query pattern
Solution Approach 2:
The patent performs preliminary aggregation calculations on history data and caches the results in advance. When real-time queries occur, the system retrieves pre-computed history data aggregation results and combines them with incremental real-time data, avoiding redundant full-data recalculations
2Productivity
If Partition Cache caches read-only partitions to improve query performance, then offline queries benefit, but real-time queries requiring daily data still need full recalculation
Solution Approach 1:
The system performs preliminary aggregation on history data and caches results partitioned by date. When querying real-time data for the current day, it retrieves pre-computed aggregation results for previous days from cache and only performs incremental aggregation on the current day's data, significantly reducing recalculation time
Solution Approach 2:
Instead of recalculating all data, the system performs partial aggregation only on the incremental real-time data portion, combining it with cached history data aggregation results. This partial action approach dramatically reduces computation while maintaining query accuracy
3Productivity
If Doris partitions are set in hours to enable real-time caching, then query concurrency improves, but metadata on FE node becomes overwhelming
Solution Approach 1:
The patent segments the partitioning strategy by data time range rather than creating hourly partitions. It caches aggregation results for history data at a coarser granularity (e.g., daily partitions) and combines with incremental real-time data, avoiding the metadata overhead of fine-grained hourly partitions while maintaining real-time query performance
Data Source
AI summary
A data query method based on Doris includes: acquiring history data in a bottom operator in a storage engine architecture of the Doris, and performing an aggregation calculation on the history data to obtain an aggregation result of the history data; caching, by a preset cache system, the aggregation result of the history data; performing an aggregation calculation on real-time data in a newly inserted bottom operator in the Doris, in response to a data query request, to obtain an aggregate result of the real-time data; acquiring the aggregation result of the history data from the preset cache system, and merging the aggregation result of the history data with the aggregation result of the real-time data to obtain merged data; and sending, by a data sending operator in the Doris, the merged data to an upper layer of an execution engine, to output a final data query result.