Sharded Bitmap Indexing for Approximate Constraint Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems struggle with handling approximate constraints, which are constraints that hold for most data tuples but fail for a small set of exceptions, leading to inefficiencies in query processing and update operations.
Innovation Solution
The PatchIndex system uses a sharded bitmap data structure to maintain tuples that satisfy and do not satisfy constraints, allowing separate processing of these two sets, and supports efficient insert, modify, and delete operations through vectorized and parallelized bulk delete operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If classical database constraints are used to ensure data integrity, then query optimization and data integrity are improved, but the system cannot handle real-world datasets with approximate constraints and exceptions
Solution Approach 1:
The patent segments the database constraint handling into two separate collections: a constraint collection for tuples satisfying the constraint and an exception collection for tuples violating the constraint. This segmentation allows the system to maintain classical constraint-based optimization for the majority of data while separately handling exceptions, thereby achieving both data integrity and adaptability to real-world datasets with approximate constraints.
2Productivity
If perfect constraints are enforced in real-world datasets, then query optimization is improved, but many update operations must be aborted due to exceptions
Solution Approach 1:
The patent extracts exception tuples from the main constraint-based processing path and places them in a separate exception collection. This allows the system to apply constraint-based query optimization to the constraint collection (containing most tuples) while separately processing exceptions, thereby maintaining high query processing efficiency while reducing update operation aborts by handling exceptions explicitly rather than failing completely.
3Speed
If constraint-based query optimization is applied to datasets with exceptions, then query performance is improved, but the index structure becomes complex to maintain during updates
Solution Approach 1:
The patent implements a dynamic index structure that automatically adapts to data changes by maintaining both constraint and exception collections. When updates occur, the system dynamically determines whether modified tuples belong in the constraint or exception collection, allowing the index to flexibly accommodate changing data while preserving constraint-based optimization benefits and avoiding rigid complex structures.
4Ease of operation
If approximate constraints are handled by excluding exception tuples, then query processing is simplified, but valuable information in exception tuples is lost
Solution Approach 1:
The patent introduces an exception collection as an intermediary structure that captures and stores exception tuples separately from the main constraint collection. This intermediary allows the system to simplify constraint handling by processing only the constraint collection for standard operations while preserving exception tuple information in the exception collection, preventing information loss and enabling selective processing based on query requirements.
Data Source
AI summary
Aspects described herein relate to maintaining a dataset with approximate constraints including determining, for a dataset, a constraint collection of tuples that satisfy a constraint and an exception collection of tuples that are an exception to the constraint, constructing, for the dataset, a sharded bitmap of bits, wherein each bit in the sharded bitmap indicates whether a tuple in the dataset is in the exception collection of tuples, wherein the sharded bitmap includes, for each shard of multiple shards, a bitmap of bits and a starting bit location index within the sharded bitmap of bits for the shard, and processing a query on the dataset including processing the constraint collection of tuples and the exception collection of tuples based on the sharded bitmap.