Virtual Data Source Manager for Big Data Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The current handling of big data is inefficient, with only a small percentage of data being utilized for decision-making due to inaccessible 'dark' data, leading to significant economic losses and increased costs from data copying and storage, which also poses security risks and performance issues.
Innovation Solution
The implementation of a virtual data source manager (VDSM) that manages virtual data sources (VDSs) using data representations like bitmap indexes, allowing for data abstraction and access without physical copying, thereby reducing data movement and storage needs, and enabling efficient data access across different locations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is copied from original data repositories to analytical data lakes and OLAP cubes for improved analytical performance, then data accessibility and analysis speed are improved, but storage costs, bandwidth consumption, and security risks increase significantly
Solution Approach 1:
The patent uses virtualization to create virtual copies of data sources that can be accessed without physically copying the actual data. Virtual data sources represent the original data through metadata and access paths, enabling analytical operations to proceed as if data were locally available while eliminating the need for physical data duplication across multiple repositories
Solution Approach 2:
The patent introduces a virtualization layer as an intermediary between original data sources and analytical workloads. This virtual data source layer acts as a mediator that provides unified access to disparate data sources without requiring data movement, thereby improving analytical performance while avoiding the storage and security issues associated with physical data copying
2Speed
If data is pre-processed, pre-joined, and aggregated into standardized formats to improve query performance, then analytical processing speed is improved, but data movement requirements and processing complexity increase
Solution Approach 1:
The patent applies preliminary action by pre-defining virtual data sources with standardized schemas and access paths before actual queries are executed. The virtualization layer pre-processes data access requests by translating queries into appropriate operations on underlying data sources, eliminating the need for repeated ETL processes and improving query performance without proportionally increasing processing complexity
Solution Approach 2:
The virtual data source layer provides universal access to multiple disparate data sources through a unified interface. It can represent various data formats and structures in a standardized way, enabling analytical tools to query different sources without custom processing logic for each source type, thereby improving speed while managing complexity
3Ease of operation
If data is copied across multiple geographic repositories to fulfill jurisdictional requirements and improve access latency, then data accessibility is improved, but security risks and storage costs increase
Solution Approach 1:
The patent uses virtual copying to create location-independent representations of data sources. Virtual data sources can be accessed from any geographic location without physically replicating data across jurisdictions, maintaining security while providing easy access. The virtualization layer handles location-specific requirements through configuration rather than data duplication
4Productivity
If batch ETL processes are used to extract, transform, and load data into analytical formats, then data usability for analysis is improved, but processing time and resource consumption increase
Solution Approach 1:
The patent enables continuous data accessibility through virtualization, eliminating the stop-start nature of batch ETL processes. Virtual data sources provide continuous access to data in usable formats without requiring periodic extraction and loading cycles, maintaining high data usability while eliminating the time losses associated with batch processing intervals
Data Source
AI summary
Techniques are disclosed relating to managing virtual data sources (VDSs), including creating and using VDSs. A virtual data source manager (VDSM) that is executing on a computer system may receive a VDS creation request to create a particular VDS. The VDS creation request may include information that is usable to identify a data origin of the particular VDS and an ingestion routine. The VDSM may use the ingestion routine to create the particular VDS that includes a bitmap index that is representative of data stored at the origin. In response to a data request corresponding to the origin, the particular VDS may be used to initiate a query to the particular VDS for data without accessing the origin. The VDSM may further maintain various software plugins that are executable to perform corresponding functions in relation to the VDSM and the VDSs managed by the VDSM.