Pipeline Control Server Data Broker Scheduling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In 1:N relationship-based data pipelines, data transmission is affected by the location of data sources and sinks, leading to increased network costs and inefficiencies, with real-time execution and resource utilization challenges due to irregular data transmission intervals and the need for batch processing.
Innovation Solution
A pipeline control server that detects registration events and schedules data transmission by provisioning a data broker with a message queue between data sources and sinks, optimizing resource usage and ensuring real-time data transmission through dynamic provisioning and scheduling based on parameter information and operation events.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a 1:N relationship-based data pipeline is used with multiple data sinks, then data transmission capability is improved, but network cost increases due to different locations of data sources and sinks
Solution Approach 1:
The patent introduces a data broker as an intermediary component between the data source and multiple data sinks. The data broker receives data from the data source and distributes it to multiple data sinks, eliminating the need for direct connections between the data source and each sink. This intermediary approach reduces network cost by consolidating data transmission paths while maintaining the 1:N relationship capability.
2Speed
If streaming execution is used for irregular data transmission intervals, then real-time data processing is improved, but computing resource waste increases due to idle pipeline periods
Solution Approach 1:
The patent implements dynamic pipeline execution by monitoring data arrival events and triggering pipeline execution only when data is available. The system transitions between active and idle states dynamically, adjusting resource allocation based on actual data transmission needs. This event-driven approach maintains real-time processing capability while eliminating waste from continuous execution during idle periods.
3Adaptability or versatility
If multiple data pipelines are managed simultaneously, then data transmission coverage is improved, but computing resource requirement increases significantly
Solution Approach 1:
The patent merges multiple data pipeline operations into a single centralized data broker that handles routing to multiple data sinks. Instead of maintaining separate pipelines for each sink, the system consolidates the routing logic and data flow management in one broker, significantly reducing the computing resources required while maintaining the ability to transmit data to multiple destinations.
4Loss of energy
If time batch scheme is used to manage multiple data pipelines, then computing resource efficiency is improved, but real-time execution guarantee is lost
Solution Approach 1:
The patent implements a feedback mechanism where the data broker monitors data arrival events in real-time and triggers pipeline execution based on actual data availability. This event-driven feedback loop ensures that data processing occurs immediately when data arrives, guaranteeing real-time execution while maintaining resource efficiency by avoiding execution during idle periods. The system adapts its execution timing based on feedback from the data stream.
Data Source
AI summary
Provided are a server for controlling data transmission through a data pipeline, and an operation method thereof. A pipeline control server is configured to create and dispose a data broker including a message queue, between a data source and a data sink according to a registration event of the data sink, and to provision an infra of the data pipeline, and an operation method thereof. A pipeline control server is configured to schedule an operation time point of a data dispersal pipeline that transmits data to a data sink based on an operation event of a data ingestion pipeline that transmits data from a data source to a data broker, and an operation method thereof.