Bitmap Index for Ad Hoc Query Speed in HBase
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing user information table systems, such as those using HBase, face performance issues and low query speed when performing ad hoc queries based on attribute values due to the lack of indexing for attribute values, requiring significant modifications to the database source code and impacting data insertion and deletion processes.
Innovation Solution
Implementing a bitmap index and user identifier array based on the user information table, allowing for efficient ad hoc queries without modifying the database or altering the original data insertion and deletion processes, by creating bitmap vectors corresponding to attribute values and using these to query user data efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a second-level index table is created in HBase to enable ad hoc queries based on attribute values, then query speed is improved, but device complexity and difficulty of maintenance increase due to significant modifications of database source code
Solution Approach 1:
The patent introduces a bitmap index as an intermediary data structure between the user information table and query operations. This bitmap index acts as a mediator that enables fast ad hoc queries without requiring modifications to the underlying HBase database source code. The bitmap index translates attribute value queries into efficient bitwise operations, resolving the contradiction between query speed improvement and system complexity increase.
Solution Approach 2:
The patent segments the indexing function into a separate bitmap index structure that is independent of the main HBase database. By creating a separate indexing layer with bitmap vectors corresponding to attribute values, the system divides the complex indexing task from the core database operations, thereby improving query performance without complicating the database source code.
2Adaptability or versatility
If a second-level index table is created using coprocessor to write index columns, then ad hoc query capability is improved, but ease of operation deteriorates because data insertion and deletion processes must be modified
Solution Approach 1:
The bitmap index serves as an intermediary layer that handles ad hoc query operations without interfering with the original data insertion and deletion processes in HBase. This mediator approach allows the system to gain versatile ad hoc query capability while keeping the core database operations simple and unchanged.
Solution Approach 2:
The patent performs preliminary indexing by creating bitmap vectors that correspond to attribute values before query operations. This preliminary action prepares the data in a format that enables fast ad hoc queries without requiring modifications to ongoing data insertion and deletion workflows, thereby maintaining ease of operation.
3Reliability
If row-by-row querying is performed based on user identifiers in HBase, then reliability is maintained, but productivity decreases due to low query speed when tens of thousands of rows are involved
Solution Approach 1:
The patent replaces the mechanical row-by-row scanning process with a bitmap-based indexing system. Instead of sequentially checking each row in HBase, the system uses pre-computed bitmap vectors to directly identify matching records through efficient bitwise operations, thereby maintaining reliability while dramatically improving query throughput for large datasets.
Solution Approach 2:
The system performs preliminary indexing by creating bitmap representations of attribute values before queries are executed. This preliminary action transforms the data into a format that enables fast retrieval without compromising the reliability of the original HBase storage, resolving the contradiction between reliable sequential access and high-speed parallel querying.
Data Source
AI summary
An information processing method and apparatus, where the method includes: obtaining a user information table; creating, based on the user information table, a bitmap index including a plurality of bitmap vectors; creating a user identifier array based on the user information table; receiving a query request sent by a client; querying the user information table based on the bitmap index, the user identifier array, and at least one attribute value, to obtain a query result; and sending the query result to the client. In the foregoing manner, second-level indexing for an ad hoc query can be implemented.