Roaring Bitmap B-Tree Indexing for Virtual Data Sources
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data management systems face inefficiencies in accessing and processing large volumes of data due to the need for extensive data copying, which increases storage costs, security risks, and query performance issues, especially when dealing with big data and diverse data sources.
Innovation Solution
The implementation of a virtual data source manager (VDSM) that manages virtual data sources (VDSs) using bitmap indexes, allowing data to be represented and accessed in a scalable and efficient manner, reducing the need for physical data duplication and enhancing query performance through data virtualization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is copied from original data repository to analytical data lakes and OLAP cubes for improved query performance, then data accessibility and analysis speed are improved, but storage costs and bandwidth consumption increase significantly
Solution Approach 1:
The patent applies copying by creating virtual copies of data through bitmap indexes rather than physical data duplication. These bitmap indexes are compact representations that can be rapidly copied and distributed across multiple locations without the bandwidth and storage overhead of copying actual data records, thus maintaining query performance improvements while minimizing storage volume increase
Solution Approach 2:
The patent changes the parameter of data representation from full data records to compressed bitmap indexes. This parameter change enables the same analytical queries to be executed on compact index structures rather than full data sets, dramatically reducing the quantity of data that needs to be stored and transmitted while preserving query functionality
2Ease of operation
If data is pre-processed and aggregated into data lakes and materialized views for faster analysis, then data accessibility is improved, but the complexity of data management and processing time increase
Solution Approach 1:
The patent implements self-service through automatic bitmap index generation and maintenance. The system automatically creates, updates, and manages bitmap indexes based on query patterns without requiring manual data lake creation or complex ETL pipeline management, thereby improving data accessibility while reducing management complexity
Solution Approach 2:
The patent introduces bitmap indexes as an intermediary layer between original data sources and analytical queries. This intermediary structure simplifies data management by providing a standardized access mechanism that works across diverse data sources without requiring complex data lakes, materialized views, or ETL processes
3Reliability
If extensive data copying is performed across multiple repositories for jurisdictional requirements and geographic distribution, then data availability and security compliance are improved, but security risks and storage costs increase
Solution Approach 1:
The patent applies copying by distributing compact bitmap indexes across multiple geographic locations and repositories to meet jurisdictional requirements. Because these indexes are compressed representations rather than full data copies, the security risk is reduced while maintaining data availability and compliance with local regulations
Solution Approach 2:
The patent segments data access functionality by creating separate bitmap indexes for different geographic regions and jurisdictions. Each location can have its own optimized index structure, allowing local query processing without requiring access to sensitive source data, thereby improving availability while minimizing security risks
Data Source
AI summary
Techniques are disclosed relating to managing virtual data sources (VDSs), including creating and using VDSs. A virtual data source manager (VDSM) that is executing on a computer system may receive a request to generate a bitmap index for a dataset. The VDSM may then generate a bitmap index by ingesting the dataset into a data format of the bitmap index. The VDSM may further generate the bitmap index by performing a compression procedure on the ingested dataset to generate a plurality of data containers, where a given data container includes a respective compressed portion of the ingested dataset. After compressing the ingested dataset, the VDSM may then store the plurality of data containers in a set of binary trees (b-trees), where the set of b-trees is usable to respond to data requests for data of the bitmap index.