Virtual Table Mapping for Schema-Free Cloud Data Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Replicating data from on-premises environments with heterogeneous data sources to cloud-based systems is challenging due to the need for extensive knowledge of the cloud environment's database structure and the requirement for matching pre-existing schemas, making current ETL solutions inefficient and resource-intensive.
Innovation Solution
A method and system that utilize virtual tables to replicate data by creating a virtual table definition and mapping it to target data types, allowing dynamic table creation in the cloud environment without requiring a pre-existing schema, and supporting metadata and content transformation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional ETL solutions are used to replicate data from on-premises environments to cloud-based systems, then data replication can be achieved, but extensive knowledge of the cloud environment's database structure is required and pre-existing schemas must be matched, making the process inefficient and resource-intensive
Solution Approach 1:
The patent introduces a cloud connector as an intermediary component that automatically manages the data replication process. This connector handles the complex tasks of schema matching, data transformation, and cloud environment interaction, eliminating the need for users to have extensive knowledge of cloud database structures. The connector acts as a mediator between the on-premises data sources and the cloud data warehouse, simplifying the ETL pipeline configuration while maintaining high replication efficiency.
2Ease of operation
If traditional ETL solutions require matching pre-existing schemas in the cloud hosted data warehouse, then data can be transformed and loaded, but the process becomes time-consuming and requires extensive knowledge of the cloud environment's database structure
Solution Approach 1:
The patent implements preliminary action by pre-configuring the cloud connector with the necessary knowledge of cloud environment schemas and data transformation rules. The connector is pre-prepared to automatically match on-premises data sources with cloud data warehouse schemas without requiring users to manually configure these mappings. This preliminary setup enables rapid deployment of data replication pipelines, significantly reducing both the time required for configuration and the operational complexity.
3Adaptability or versatility
If on-premises environments include heterogeneous data sources with different formats and data storage schemas, then diverse data can be collected, but replicating this data to the cloud becomes a difficult and resource-intensive task
Solution Approach 1:
The patent implements universality by designing the cloud connector to handle multiple types of data sources and formats through a single unified interface. The connector is configured to automatically detect and adapt to different on-premises data source types (databases, files, applications) and their varying formats and schemas. This multi-functional capability allows the system to replicate diverse heterogeneous data to the cloud efficiently, without requiring separate configurations for each data source type, thereby reducing overall resource intensity.
Data Source
AI summary
Embodiments provide systems, methods and computer program products for cloud replication of data. One embodiment includes, accessing a virtual table definition and a data collection definition, the virtual table definition comprising a definition of a set of virtual table fields and a mapping of the set of virtual table fields to a set of target data types; automatically creating a virtual table according to the virtual table definition, the virtual table comprising the set of virtual table fields populated with the data of interest according to the data collection definition; and sending the virtual table and the mapping to a cloud computing environment. One embodiment further includes, in the cloud computing environment, storing the set of virtual table fields as a set of physical table fields in a physical table in a cloud hosted database, the set of physical table fields having the set of target data types.