Remote Data Lake Relational Dataset Construction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing cloud computing systems face challenges in managing relational databases due to limitations in ingesting and maintaining data consistency, particularly in data lake environments, which leads to issues with latency, resource utilization, and the need for specialized expertise.
Innovation Solution
A method is introduced to construct a relational database within a remote computing environment using functional models, enforcing foreign key relationships and utilizing ETL tools to manage data integration, thereby facilitating the ingestion of relational data and improving data consistency and performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If cloud computing systems store data in disparate locations without relational structure, then data storage capacity and flexibility are improved, but data consistency and query performance deteriorate
Solution Approach 1:
The system segments data storage into two distinct layers: a relational database layer that maintains structured data with foreign key relationships, and a cloud data lake layer that stores unstructured or semi-structured data. This segmentation allows each layer to optimize for its specific purpose while working together through defined interfaces.
Solution Approach 2:
The patent introduces an intermediary component that acts as a bridge between the relational database and the cloud data lake. This intermediary handles data synchronization, relationship enforcement, and query coordination, ensuring that foreign key relationships are maintained even as data moves between the structured and unstructured storage environments.
2Device complexity
If cloud computing systems lack native relational database support, then infrastructure complexity is reduced, but functionality for managing relational data deteriorates
Solution Approach 1:
The system merges the capabilities of a traditional relational database with cloud data lake functionality by implementing foreign key relationship enforcement within the cloud environment. This combination allows the system to maintain the data integrity features of relational databases while leveraging the scalability and flexibility of cloud storage.
Solution Approach 2:
The patent creates a universal data management system that can handle both structured relational data and unstructured cloud data within a single platform. The system provides multi-functional capabilities including relational query processing, data lake storage, and automated relationship maintenance, eliminating the need for separate specialized systems.
3Productivity
If cloud computing systems ingest large amounts of data without relational constraints, then data ingestion speed is improved, but data quality and error rates worsen
Solution Approach 1:
The system performs preliminary actions by establishing foreign key relationships and data constraints before data is fully ingested into the cloud environment. This proactive approach ensures that data quality rules are enforced during the ingestion process itself, preventing poor quality data from being loaded in the first place while maintaining high ingestion speeds.
Data Source
AI summary
A system, device and method are provided for managing data ingested into remote computing environments. The illustrative method includes ingesting a first set of data into a first container of a remote computing environment (RCE), the ingesting resulting in a data set structured with structural formatting of the RCE. The method includes determining a functional data model applicable to the ingested data set, and constructing, in the RCE, a relational dataset by at least in part applying the functional data model. The method includes generating a base data set from the ingested data set and the relational data set and persisting the base data set in a second container, the relational data set contributing via one or more tools for accessing relational databases. The method includes returning at least some of the base data set to in response to user requests to access enterprise data.