Page-Based Index Query Structure for Read-Only Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing index structures for read-only scenarios in databases, such as those based on LSM-Tree, lack optimization for efficient data query operations, leading to suboptimal performance due to the need to load and parse entire index files, which results in inefficient memory usage and calculation delays.
Innovation Solution
An index query method and apparatus that optimizes index structures for read-only scenarios by using a page-based HashMap data structure with a page index file comprising an entry array, chunk bitmap, and length array, ensuring O(1) calculation complexity and high space utilization, allowing on-demand loading and lightweight parsing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional index structures are used for read-only scenarios, then the index can store all data, but the entire index file must be loaded and parsed, leading to high memory usage and calculation delays
Solution Approach 1:
The patent divides the traditional monolithic index file into multiple page index files, where each page index file corresponds to a specific data range. This segmentation allows the system to load only the necessary page index files into memory based on the query conditions, rather than loading the entire index file. The page index files are organized in a hierarchical structure with header information that enables efficient range queries and selective loading, directly reducing memory usage and query delays while maintaining data completeness.
2Ease of manufacture
If traditional index structures are used, then the index can be built once, but parsing the entire index file every time results in high calculation complexity
Solution Approach 1:
The patent pre-calculates and stores metadata information in the header of each page index file, including the minimum and maximum values of the indexed field, the number of entries, and the file size. This preliminary action eliminates the need to parse the entire index file during queries, as the system can use the header information to quickly determine which pages to access and how many entries to load. The header parsing is minimal and does not require traversing the entire index structure, significantly reducing calculation complexity while maintaining ease of index construction.
3Productivity
If the entire index file is loaded into memory, then all data can be accessed, but memory usage becomes inefficient
Solution Approach 1:
The patent segments the index file into multiple page index files that can be independently loaded into memory. Each page index file contains a manageable subset of the total index entries, allowing the system to load only the necessary pages based on the query's data range requirements. The header information in each page index file enables the system to quickly identify which pages contain the required data, ensuring full data accessibility while maintaining efficient memory usage by avoiding loading unnecessary index entries.
4Adaptability or versatility
If traditional HashMap structures are used, then general capabilities are provided, but they lack optimization for read-only scenarios
Solution Approach 1:
The patent optimizes the HashMap structure specifically for read-only scenarios by implementing a page-based organization where each page index file is designed with header information containing metadata about the data range, entry count, and file size. This local optimization allows the system to quickly determine which pages to access without needing to traverse the entire structure, significantly improving query efficiency for read-only operations while maintaining the general HashMap capabilities for insertion, deletion, and lookup operations.
Data Source
AI summary
The present application discloses an index query method, apparatus, and device based on a database, wherein index optimization is performed for a read-only scenario. The method includes: determining a page identification corresponding to a target key in a data query request based on a hash value of the target key; obtaining a page index file corresponding to the page identification, wherein the page index file has a page index structure, the page index structure includes an entry array, a chunk bitmap and a length array, the entry array includes entry data of the page index file, and the entry data is sorted in an order of buckets to which the entry data belong and in an order of the entry data in the buckets; and querying, in the page index file, whether there is a value corresponding to the target key.