Adaptive Storage Layering for Cloud OLAP Query Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current cloud computing-based big data architectures face limitations in query response times due to bandwidth bottlenecks and inefficient caching mechanisms, particularly in distributed file cache systems like Alluxio, which struggle with high concurrency and lack elastic expansion, leading to slow initial query speeds and high costs.
Innovation Solution
A cloud computing-based adaptive storage layering system that includes a data node management module for horizontal expansion and shrinkage, a metadata management module for aggregating query data, an adaptive storage layering module for pre-loading and caching, and a pre-aggregated query routing module for dynamic query address switching, optimizing storage and computing resources based on cache conditions and model hit counts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is stored on cloud storage with separation of storage and computing, then cost performance is enhanced and computing resources can be dynamically adjusted, but network transmission bandwidth becomes a bottleneck limiting data analysis speed
Solution Approach 1:
The patent implements pre-computation of OLAP models and pre-loading of hot data into cache before actual queries occur. The system proactively identifies frequently accessed data patterns and prepares them in advance, so when queries arrive, the data is already ready in cache rather than needing to be fetched from cloud storage over the network.
Solution Approach 2:
The patent introduces a distributed cache system as an intermediary layer between cloud storage and computing nodes. This cache acts as a buffer that stores frequently accessed data locally, eliminating the need for repeated network transmissions between cloud storage and computing nodes, thus resolving the bandwidth bottleneck while maintaining the storage-computing separation architecture.
2Adaptability or versatility
If distributed file cache systems like Alluxio are used to support multiple clouds, then multi-level storage is enabled, but cache replacement algorithms are too simple and cannot achieve elastic expansion for pre-computing scenes
Solution Approach 1:
The patent implements dynamic cache management where the cache capacity and configuration can automatically expand and contract based on workload demands. The system monitors query patterns and data access frequencies in real-time, dynamically adjusting which data resides in cache and for how long, enabling the cache to adapt elastically to changing pre-computing requirements rather than using fixed, simple replacement algorithms.
Solution Approach 2:
The patent employs feedback mechanisms that continuously monitor cache hit rates, query patterns, and data access frequencies. Based on this feedback, the system intelligently adjusts cache replacement policies and pre-computation priorities, allowing the cache system to learn from actual usage patterns and optimize itself, rather than relying on static, simple replacement algorithms.
3Productivity
If a cluster is stopped in idle and started when required to reduce costs, then cloud costs are lowered, but initial query speed is very slow because model index files cannot be dynamically preheated
Solution Approach 1:
The patent implements pre-warming of the cache system and pre-loading of model index files during the cluster startup process. Before the cluster becomes fully operational, the system proactively loads frequently accessed data and model indexes into cache, so when the first queries arrive, the data is already ready rather than needing to be fetched from cloud storage, thus eliminating the cold start penalty.
Solution Approach 2:
The patent implements periodic cache warming and data pre-loading operations that occur at regular intervals or based on predicted workload patterns. Even during idle periods, the system periodically refreshes and pre-loads data into cache, ensuring that when the cluster is activated, the cache is already warm and ready to handle queries immediately, thus reducing initial query latency while maintaining cost savings during truly idle periods.
Data Source
AI summary
The disclosure provides a cloud computing-based adaptive storage layering system and method. The system includes a data node management module, a metadata management module, an adaptive storage layering module, and a pre-aggregated query routing module. According to predefined rules, node capacity is expanded and shrunk, collected queried hit models and scanned file paths are aggregated and sorted, and layering loading and pre-loading are performed on files. Based on the disclosure, an efficient On-Line Analytical Processing (OLAP) query execution engine may be constructed, to deal with complex OLAP queries of various reporting system. Therefore, the execution efficiency of cloud OLAP engines can be significantly enhanced.