Virtual Data Source Manager Bitmap Index Architecture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The current handling of big data leads to inefficiencies due to the lack of access to vast amounts of data, resulting in unrealized potential business value and high costs associated with data copying and storage, which is bandwidth-intensive and increases security risks.
Innovation Solution
The implementation of a virtual data source manager (VDSM) that manages virtual data sources (VDSs) using data representations like bitmap indexes, allowing for efficient data access and processing without the need for physical data copies, by creating, managing, and linking VDSs across different locations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is copied from original data repositories to analytical data lakes and OLAP cubes for improved data accessibility and analytical performance, then data access efficiency is improved, but hardware footprint and storage costs increase significantly
Solution Approach 1:
The patent uses bitmap indexes as compact data representations that serve as virtual copies of the original data. These bitmaps are dramatically smaller than traditional data copies while still enabling efficient analytical queries. The bitmap indexes are stored in a data lake and can be queried directly without accessing the original large datasets, thus improving data access efficiency while minimizing hardware footprint requirements.
Solution Approach 2:
The patent transforms data into a different representation format (bitmap indexes) that changes the fundamental parameters of data storage. Instead of storing actual data values, the system stores bitmaps where each bit represents the presence or absence of a data value. This parameter transformation reduces storage requirements by orders of magnitude while maintaining query capability, directly resolving the contradiction between accessibility and hardware footprint.
2Productivity
If data is pre-processed, pre-joined, and aggregated into formatted structures for analytical purposes, then analytical performance is improved, but data movement complexity and bandwidth requirements increase
Solution Approach 1:
The patent performs preliminary action by pre-computing bitmap indexes from the original data sources and storing them in the data lake. This pre-processing transforms the data into an analytical-optimized format (bitmaps) that can be directly queried without requiring complex ETL processes at query time. The bitmap indexes are pre-formed and ready for immediate analytical use, improving performance while reducing the complexity of data migration and preparation.
Solution Approach 2:
The patent extracts the essential analytical information from the original data by creating bitmap representations. Instead of copying and transforming entire datasets through complex ETL processes, the system extracts only the necessary information into compact bitmap form. This extraction approach maintains analytical performance while dramatically reducing data movement complexity and bandwidth requirements.
3Ease of operation
If multiple copies of data are stored across different geographic regions to fulfill jurisdictional requirements and improve access performance, then data accessibility is improved, but security risks and storage costs increase
Solution Approach 1:
The patent creates compact bitmap copies of data that can be distributed to multiple geographic regions. Because bitmap indexes are so much smaller than traditional data copies, the security risks and storage costs associated with multiple copies are significantly reduced. The bitmaps can be securely stored in distributed data lakes across different regions, providing both jurisdictional compliance and improved access performance without the security vulnerabilities of traditional data replication.
4Adaptability or versatility
If original data is extracted in batch processes and transformed into ETL formats for loading into analytical systems, then data analytical usability is improved, but processing time and bandwidth consumption increase
Solution Approach 1:
The patent changes the fundamental parameter of data representation from traditional ETL formats to bitmap indexes. This parameter transformation enables direct analytical queries on the transformed data without requiring complex batch processing and formatting operations. The bitmap structure is inherently optimized for analytical operations, allowing the system to achieve high analytical usability with minimal processing time and bandwidth consumption.
Data Source
AI summary
Techniques are disclosed relating to managing virtual data sources (VDSs), including creating and using VDSs. A virtual data source manager (VDSM) that is executing on a computer system may receive a VDS creation request to create a particular VDS. The VDS creation request may include information that is usable to identify a data origin of the particular VDS and an ingestion routine. The VDSM may use the ingestion routine to create the particular VDS that includes a bitmap index that is representative of data stored at the origin. In response to a data request corresponding to the origin, the particular VDS may be used to initiate a query to the particular VDS for data without accessing the origin. The VDSM may further maintain various software plugins that are executable to perform corresponding functions in relation to the VDSM and the VDSs managed by the VDSM.