RocksDB Ordered View Cache for Query Speed and CPU Usage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing data query methods in RocksDB, such as the heap-based approach, lead to random I/O operations and resource wastage due to the need for reconstructing ordered views, which slows down querying speed and increases CPU usage.
Innovation Solution
A data query method that utilizes a main cache to store historical ordered views, allowing for efficient querying of keywords by first checking the cache and then the hard disk and memory, reducing unnecessary I/O operations and resource wastage by storing and retrieving data based on keyword access patterns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the heap-based approach is used to query data in RocksDB, then the querying can traverse multiple SST files, but this leads to random I/O operations and increased CPU usage, slowing down querying speed
Solution Approach 1:
The patent pre-builds ordered views from SST files and stores them in the main cache before queries occur. This preliminary action allows the system to have ready-to-use ordered data structures, eliminating the need to construct heaps and traverse multiple SST files during actual queries, thereby reducing CPU usage and improving query speed
Solution Approach 2:
The patent introduces ordered views as an intermediary data structure between the raw SST files and the query processing mechanism. These ordered views serve as a pre-processed intermediate layer that simplifies query operations, reducing both CPU overhead and I/O operations by providing a directly queryable structure
2Adaptability or versatility
If the heap-based approach reconstructs ordered views after querying, then the ordered view can be used again in subsequent accesses, but this causes waste of resources due to unnecessary reconstruction
Solution Approach 1:
The patent maintains ordered views continuously in the main cache after they are built, allowing them to be reused across multiple queries without reconstruction. This continuous availability eliminates the cycle of building and discarding ordered views, reducing resource wastage while maintaining adaptability for subsequent accesses
Solution Approach 2:
The patent implements a mechanism to detect when ordered views are still valid and can be reused, avoiding unnecessary reconstruction. By recovering and reusing existing ordered views in the main cache rather than discarding them, the system reduces resource wastage while maintaining the ability to adapt to subsequent query needs
3Productivity
If the system queries data by traversing multiple SST files at multiple levels, then comprehensive data can be accessed, but this involves a number of I/O operations that slow down querying
Solution Approach 1:
The patent pre-processes SST files to build ordered views that contain the necessary data organized in an efficient query structure. This preliminary organization of data eliminates the need for multi-level traversal during queries, reducing I/O operations and querying time while maintaining complete data access capability
Solution Approach 2:
The patent extracts the essential queryable information from SST files and stores it in ordered views in the main cache. By taking out only the necessary data and organizing it in an optimized structure, the system reduces the amount of I/O required during queries while preserving the ability to access comprehensive data
Data Source
AI summary
Disclosed are a data query method, a data query device, a storage device, and an electronic device. The data query method comprises: receiving a data query request carrying a keyword associated with data to be queried; querying the keyword in the main cache, wherein ordered views that have been accessed historically are stored in the main cache and each of the ordered views comprises a plurality of historical keywords; in a case that a historical keyword matching the keyword is not found in the main cache, querying the data corresponding to the keyword in the hard disk and the memory based on the keyword and obtaining the data to be queried; and in a case that a historical keyword matching the keyword is found in the main cache, querying the data corresponding to the keyword in the memory based on the keyword and obtaining the data to be queried.