Virtual Data Source Manager for Big Data Analytics

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

The current handling of big data is inefficient, with only a small percentage of data being utilized for decision-making due to inaccessible 'dark' data, leading to significant economic losses and increased costs from data copying and storage, which also poses security risks and performance issues.

Innovation Solution

The implementation of a virtual data source manager (VDSM) that manages virtual data sources (VDSs) using data representations like bitmap indexes, allowing for data abstraction and access without physical copying, thereby reducing data movement and storage needs, and enabling efficient data access across different locations.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If data is copied from original data repository to analytical data lakes and OLAP cubes for improved analytical performance, then data accessibility and analysis speed are improved, but storage costs and bandwidth consumption increase significantly

Engineering Contradiction:

Improvedata analysis speedVSAvoiddata storage volume

Core Design Contradiction:

ProductivityVSQuantity of substance

Solution Approach 1:

The patent applies copying by creating virtual data source copies that reference original data without duplicating the actual data bytes. Virtual data sources are created from original data sources, allowing multiple analytical systems to access the same underlying data through virtual interfaces, eliminating the need for physical data duplication across data lakes and OLAP cubes.

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The patent introduces virtual data sources as intermediary layers between original data sources and analytical systems. These virtual data sources act as mediators that provide unified access interfaces to multiple analytical applications, allowing them to query and analyze data without requiring physical data copies, thus reducing storage requirements while maintaining analytical capabilities.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Speed

If data is pre-processed, pre-joined, and aggregated into formatted structures for better analytical performance, then query speed and ease of analysis are improved, but data processing time and computational resources increase

Engineering Contradiction:

Improvequery execution speedVSAvoiddata processing time

Core Design Contradiction:

SpeedVSLoss of time

Solution Approach 1:

The patent applies preliminary action by pre-defining virtual data source structures, access paths, and query templates during system setup rather than during data processing. Virtual data sources are configured with predefined schemas and access methods that enable fast query execution without requiring extensive real-time data processing or transformation operations.

Inventive Principle:

Principle #10Preliminary action

3Ease of operation

If multiple copies of data are stored across different geographic regions to fulfill jurisdictional requirements and improve access performance, then data accessibility and compliance are improved, but security risks and storage costs increase

Engineering Contradiction:

Improvedata access convenienceVSAvoiddata security

Core Design Contradiction:

Ease of operationVSReliability

Solution Approach 1:

The patent applies copying by creating virtual data source references that can be replicated across multiple geographic locations without duplicating actual data. Virtual data sources are lightweight metadata structures that point to original data, allowing them to be copied and distributed freely while the actual data remains stored in a single secure location, thus eliminating security risks associated with data replication.

Inventive Principle:

Principle #26Copying

4Speed

If data is moved closer to users and applications in different geographic regions, then data access performance is improved, but bandwidth consumption and data movement costs increase

Engineering Contradiction:

Improvedata access speedVSAvoidbandwidth consumption

Core Design Contradiction:

SpeedVSLoss of energy

Solution Approach 1:

The patent applies copying by distributing virtual data source metadata rather than actual data bytes. Virtual data sources are lightweight structures containing schema information and access paths that can be rapidly replicated across geographic regions with minimal bandwidth consumption, while the actual data remains stationary in its original location, eliminating the need for expensive data movement operations.

Inventive Principle:

Principle #26Copying

Data Source

PatentUS11687513B2Virtual data source manager of data virtualization-based architecture

Publication Date: 2023.06.27 CIRCUIT HOLDINGS INC

AI summary

Techniques are disclosed relating to managing virtual data sources (VDSs), including creating and using VDSs. A virtual data source manager (VDSM) that is executing on a computer system may receive a VDS creation request to create a particular VDS. The VDS creation request may include information that is usable to identify a data origin of the particular VDS and an ingestion routine. The VDSM may use the ingestion routine to create the particular VDS that includes a bitmap index that is representative of data stored at the origin. In response to a data request corresponding to the origin, the particular VDS may be used to initiate a query to the particular VDS for data without accessing the origin. The VDSM may further maintain various software plugins that are executable to perform corresponding functions in relation to the VDSM and the VDSs managed by the VDSM.