Cloud Data Pipeline Service for Secure Public-Private Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current cloud computing platforms face challenges in efficiently transferring large volumes of data from public clouds to private clouds, as existing communication models are either resource-intensive or expose private clouds to security risks, and there is a lack of efficient mechanisms for data pipelines between these platforms.
Innovation Solution
The implementation of a data pipeline service that includes a control plane gateway to register data consumers and producers, generate persistent data buffers, and establish secure data pipelines between public and private clouds, allowing for asynchronous or synchronous data transfer while maintaining network separation and security.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing communication models are used for data transfer between public and private clouds, then data transfer can be achieved, but resource consumption increases and security risks are exposed
Solution Approach 1:
The patent introduces a data pipeline service as an intermediary component that sits between the public cloud and private cloud. This service includes control plane gateways and data plane gateways that mediate data transfer, enabling secure communication without direct exposure of private cloud resources while maintaining efficient data flow through buffered asynchronous operations.
2Speed
If direct data transfer between public and private clouds is implemented, then data transfer speed improves, but security risks increase
Solution Approach 1:
The patent segments the data transfer system into distinct components: control plane gateways for management, data plane gateways for actual data flow, and buffered storage for asynchronous operations. This segmentation allows high-speed data transfer through dedicated data plane gateways while the control plane remains isolated, reducing security risks through architectural separation.
Solution Approach 2:
Data plane gateways act as intermediaries that handle high-speed data transfer between public and private clouds without exposing private cloud infrastructure. The buffered architecture further mediates data flow, allowing asynchronous operations that maintain speed while controlling security exposure.
3Reliability
If resource-intensive communication models are used, then data transfer reliability improves, but efficiency decreases
Solution Approach 1:
The patent implements buffered asynchronous data transfer where data is transferred in periodic batches rather than continuous real-time streams. This periodic action through buffers maintains data transfer reliability through controlled pacing while improving overall efficiency by reducing the overhead of continuous connection management and resource allocation.
Data Source
AI summary
Methods, apparatus, systems and articles of manufacture to establish a data pipeline between cloud computing platforms. An apparatus includes at least one memory, machine readable instructions in the apparatus, and processor circuitry to execute the machine readable instructions to at least extract a data producer name from data, the data to be provided from a data producer to a data consumer, identify a buffer identifier based on a mapping of the data producer name to the buffer identifier, cause transmission of the data to a buffer associated with the buffer identifier, and cause transmission of the data from the buffer to the data consumer based on an association between the buffer identifier and a data consumer name, the data consumer name corresponding to the data consumer.