Columnar Database Bitmaps for Missing-Row Write Performance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing columnar storage databases face reduced write performance due to the need for query and completion processes when data loss occurs, particularly in scenarios where incomplete rows are written, and existing read optimizations are hindered by data loss during predicate execution.
Innovation Solution
Implement a data processing method and apparatus that uses bitmaps to store line loss identification for each column, aligning actual data with loss identification columns in a memory structure, allowing direct data reading and eliminating the need for query and completion processes during writing, and adjusting index types to handle data loss during predicate execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is stored in a traditional row-based format in columnar storage database, then write operations are simple and direct, but read operations require query and completion processes that reduce write performance when data loss occurs
Solution Approach 1:
The patent segments the data storage structure into column-based units with independent bitmaps for each column. Each column stores its data and associated bitmap separately, allowing independent processing of columns during read operations. This segmentation eliminates the need for row-based query and completion processes, directly improving write performance while maintaining data integrity tracking through bitmaps.
Solution Approach 2:
The patent transitions from traditional row-based storage to column-based storage, adding a dimensional shift in data organization. By storing data vertically by column rather than horizontally by row, the system enables direct columnar reads and writes without requiring row assembly, thereby improving write performance and eliminating query completion overhead.
2Reliability
If bitmaps are used to track data loss for each column, then data integrity is maintained, but storage space and processing overhead increase
Solution Approach 1:
The patent uses bitmaps as thin, flexible metadata structures associated with each column. These bitmaps act as lightweight shells that track data loss status without adding substantial overhead. Each bitmap is only as large as the number of rows in its column, providing efficient space utilization while maintaining complete data integrity tracking.
3Reliability
If query and completion processes are used to handle data loss, then data completeness is ensured, but write performance is reduced
Solution Approach 1:
The patent performs preliminary action by pre-tracking data loss status using bitmaps during the write operation itself. Instead of querying and completing data after writing, the bitmap records which rows have data loss upfront, allowing the system to proceed with write operations at full speed while maintaining awareness of data completeness issues for later handling.
Solution Approach 2:
The patent extracts the data loss tracking function from the write operation itself. By separating the tracking mechanism (bitmap) from the write process, the system can write data directly without performing query and completion processes, thereby maintaining data completeness tracking while preserving write performance.
4Speed
If traditional read optimization is applied, then read speed is improved, but predicate execution is hindered by data loss
Solution Approach 1:
The patent introduces bitmaps as intermediary structures between the stored data and the read/predicate execution processes. These bitmaps serve as mediators that provide fast filtering capability for read optimization while simultaneously conveying data loss information to predicate execution, ensuring both read speed improvement and predicate execution accuracy by acting as an information bridge.
Data Source
AI summary
The present application discloses a data processing method and an apparatus based on a columnar storage database and a device, which are used for improving performance of data writing in a scenario where a database has data loss. The method comprises: storing, for a target column in a data table, a line loss identification of the target column by using a bitmap corresponding to the target column, wherein each bit of the bitmap corresponds to whether each line of the target column has data loss, and the target column is any column in the data table; storing actual data included in the target column; receiving a data read request, wherein the data read request includes a column; and establishing, in a memory, a memory structure of the column based on a bitmap corresponding to the column and actual data included in the column.