Automated Data Pipeline Configuration via Network Sensor Tagging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In IoT networks, the lack of harmonization between data consumers leads to complex data access policies, resource consumption issues, and the need for individual configuration of data pipelines for each combination of protocols and destination cloud providers, resulting in increased overhead and incompatibility across cloud providers.
Innovation Solution
A service associates descriptive tags with network nodes based on packet inspection by sensors, identifying data to be extracted and determining the external destination, and sends a data pipeline configuration to the edge device to extract and send data accordingly, enabling automated configuration of data models and management pipelines.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data pipelines are individually configured for each combination of protocols and destination cloud providers, then data access requirements are met, but configuration complexity and overhead increase significantly
Solution Approach 1:
The patent introduces an intermediary service that sits between IoT devices and cloud providers, automatically generating and managing data pipeline configurations. This service acts as a mediator that translates between different protocols and cloud provider requirements, eliminating the need for manual configuration of each pipeline while maintaining adaptability across multiple protocols and destinations
Solution Approach 2:
The patent creates a universal data pipeline configuration system that can handle multiple protocols and cloud providers through a single automated service. This universal approach allows the same infrastructure to serve multiple purposes - supporting different data formats, protocols, and destination platforms without requiring separate configuration processes for each
2Reliability
If manual configuration of data pipelines is performed for each IoT node, then precise data routing is achieved, but time and resource consumption increase
Solution Approach 1:
The patent implements a self-service mechanism where the automated service discovers IoT devices, inspects their data formats and protocols, and automatically generates appropriate pipeline configurations without human intervention. The system serves itself by autonomously managing the entire configuration lifecycle from device discovery to pipeline deployment, ensuring accurate routing while eliminating manual configuration time
Solution Approach 2:
The patent performs preliminary actions by pre-configuring template-based pipeline patterns that can be automatically applied to IoT devices. The system prepares configuration templates in advance that match common device types and protocols, allowing rapid deployment when devices are discovered. This preliminary preparation enables the system to quickly generate accurate configurations without manual intervention
3Adaptability or versatility
If multiple cloud providers are integrated with individual configurations, then data consumer requirements are satisfied, but resource consumption and overhead increase
Solution Approach 1:
The patent merges multiple cloud provider integrations into a single automated service that manages all connections centrally. Instead of maintaining separate configuration systems for each cloud provider, the patent combines them into one unified service that handles protocol translation, data formatting, and routing to multiple destinations simultaneously, reducing overall computational overhead while maintaining compatibility with all providers
Data Source
AI summary
In one embodiment, a service associates a plurality of descriptive tags with a node in a network, based on an inspection of packets sent by the node that is performed by one or more sensors deployed to the network. The service identifies, based on the plurality of descriptive tags, data to be extracted from traffic of the node by an edge device located at an edge of the network. The service determines, based on the plurality of descriptive tags, an external destination to which the data should be sent by the edge device after extraction. The service sends a data pipeline configuration to the edge device, wherein the data pipeline configuration causes the edge device to extract the data from the traffic sent by the node and to send the data to the external destination.