Telemetry Data Transport Service with Priority Queues
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current network data centers face challenges in efficiently and securely transporting telemetry data across networks, particularly in ensuring reliable, scalable, and authenticated communication between clients and back-end services while maintaining message priority and geographic location filtering.
Innovation Solution
A method and system for transporting telemetry data that involves a data transport service receiving requests from clients, authenticating them, parsing commands for storing messages in physical and virtual queues, and ensuring secure, priority-based message delivery across networks using a priority First-In First-Out (FIFO) queue system, supporting binary and text data formats, and allowing messages of any size.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If authentication credentials are verified for each telemetry data request, then security and reliability are improved, but processing time and system complexity increase
Solution Approach 1:
The system performs authentication credentials verification as a preliminary action before processing the actual telemetry data request. By authenticating the client application beforehand through the data transport service, the system ensures security requirements are met while establishing a trusted connection that can efficiently handle subsequent data transmission without repeated authentication overhead.
2Reliability
If message priority is maintained and enforced in the queue system, then service quality and reliability are improved, but system complexity and processing overhead increase
Solution Approach 1:
The queue system is segmented into multiple priority levels, allowing messages to be categorized and handled according to their importance. This segmentation enables the system to maintain different queue structures for different priority levels, ensuring that high-priority telemetry messages are processed promptly while managing overall system complexity through modular queue organization.
Solution Approach 2:
Different parts of the queue system are assigned different quality characteristics based on message priority. High-priority messages receive preferential treatment with faster processing and guaranteed delivery, while lower-priority messages are handled with standard procedures. This local quality approach ensures reliable delivery for critical telemetry data without unnecessarily complicating the entire queue management system.
3Measurement precision
If virtual queues are implemented for different client applications, then message routing accuracy and service quality are improved, but system complexity and memory usage increase
Solution Approach 1:
Virtual queues are implemented as logical copies or references to the underlying physical queue structure, rather than completely separate data structures. Each client application has its own virtual queue view that maps to specific portions or priorities within the physical queue, enabling accurate message routing and client-specific message handling while sharing the same physical storage infrastructure, thus reducing overall system complexity and memory usage.
Data Source
AI summary
A method for transporting telemetry data involves receiving a request from a client in a first network. The client is registered with a data transport service executing in a second network. The request includes an authentication credential for the client, a message including telemetry data, and a message priority. The method further involves authenticating the request using the authentication credential. The method further involves parsing the request to obtain and subsequently execute a command for storing the message in a physical queue of the second network. The physical queue is associated with a back-end service executing in the second network. A virtual queue maintains entries for messages stored in the physical queue that were sent from the first network to the second network. Another virtual queue maintains entries for messages stored in the physical queue that were sent from the second network to the first network.