Microdatabase Virtualization for Heterogeneous Data Integration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large entities face challenges with data consistency, configuration control, and access speed due to the use of multiple heterogeneous data systems with different database topologies, particularly after mergers or acquisitions, as migrating these systems is costly and time-consuming.
Innovation Solution
The approach involves creating microdatabases that store data from multiple data stores with different database topologies, using key map records to link queries to the appropriate microdatabases, and employing access keys for rapid data retrieval, allowing for efficient access and reconstruction of data even in case of system failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If multiple heterogeneous data systems are maintained separately after merger or acquisition, then data system migration cost and time are reduced, but data consistency and access speed deteriorate
Solution Approach 1:
The patent introduces a data virtualization layer as an intermediary between multiple heterogeneous data systems. This layer provides a unified access interface and implements data consistency protocols without requiring physical migration of data between systems, thus maintaining data consistency while avoiding migration time and cost.
Solution Approach 2:
The patent segments the data access architecture into independent logical units that can access different underlying data systems. Each segment can be configured to access specific data sources while maintaining unified access controls and consistency rules at the segmentation boundary.
2Loss of time
If multiple heterogeneous data systems are maintained separately after merger or acquisition, then data system migration cost and time are reduced, but access speed deteriorates
Solution Approach 1:
The data virtualization layer acts as a smart intermediary that caches frequently accessed data and optimizes query routing to underlying systems, improving access speed without requiring physical data consolidation.
Solution Approach 2:
The system performs preliminary data preprocessing and caching at the virtualization layer, so that when access requests are made, data can be retrieved faster from the cache or pre-processed results rather than querying the underlying heterogeneous systems in real-time.
3Reliability
If data is stored in large centralized data stores, then data completeness is maintained, but data access speed deteriorates from minutes to milliseconds
Solution Approach 1:
The patent segments large centralized data stores into smaller micro-databases organized by domain or function. Each micro-database can be independently accessed and optimized, significantly improving access speed while maintaining data completeness through the virtualization layer's ability to aggregate results from multiple segments.
Solution Approach 2:
The patent introduces a new dimensional organization of data along functional or domain lines rather than traditional hierarchical structures. This dimensional change enables parallel access paths and optimized query routing, achieving millisecond-level access speeds while preserving complete data availability.
Data Source
AI summary
A method includes retrieving first data from a first data store having a first database topology and second data from a second data store having a second database topology. The first data and the second data are stored in a load data set. The load data set is sorted based on a first identifier field. A first microdatabase is loaded with a first portion of the first data associated with a first value of the first identifier field and a first portion of the second data associated with the first value. A second microdatabase is loaded with a second portion of the first data associated with a second value of the first identifier field and a first portion of the second data associated with the second value. A first key map record for the first microdatabase is generated. A second key map record for the second microdatabase is generated.