Dynamic Telemetry Catalog for Cluster File Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing telemetry architectures in distributed systems are inflexible and unable to dynamically add new metric datasets, data producers, or consumers, failing to adapt to the evolving complexity of these systems.
Innovation Solution
A subscription-based telemetry system with role-based access control (RBAC) that allows dynamic registration of new metrics, enabling flexible data management through dynamic cataloging and secure access control, optimized data collection, and efficient data transmission.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a fixed telemetry architecture is used, then system stability is maintained, but adaptability to add new metrics and data producers is lost
Solution Approach 1:
The patent implements a dynamic telemetry architecture where the catalog of available metrics and data producers can be modified at runtime. New metrics and data producers can be registered without requiring system restarts or complex reconfiguration, allowing the system to adapt dynamically to changing requirements while maintaining operational stability.
Solution Approach 2:
The patent introduces a catalog service as an intermediary component that mediates between data producers and consumers. This catalog maintains metadata about available metrics and data producers, enabling flexible discovery and subscription without direct complex interactions between all system components, thus managing complexity while improving adaptability.
2Adaptability or versatility
If dynamic registration of new metrics is enabled, then adaptability is improved, but security control over data access may be compromised
Solution Approach 1:
The patent implements preliminary action through schema validation and catalog registration requirements. Before a new metric or data producer can be registered, it must define a schema that conforms to predefined templates, and the registration process itself is controlled and logged. This preliminary verification ensures security and data quality are maintained even as the system dynamically accepts new metrics.
Solution Approach 2:
The system incorporates feedback mechanisms where the catalog service validates and monitors registered metrics and data producers. Subscription requests are processed with security checks, and the system provides feedback about available metrics to consumers. This feedback loop ensures that only authorized and validated metrics can be accessed, maintaining security while enabling dynamic registration.
3Quantity of substance
If all telemetry data is collected, then data completeness is improved, but network bandwidth consumption increases
Solution Approach 1:
The patent segments telemetry data collection through the subscription model, where consumers explicitly subscribe to only the metrics they need. The catalog service divides available metrics into distinct categories, and the system collects and transmits only the segmented data that subscribed consumers require, avoiding unnecessary network traffic while maintaining data completeness for authorized users.
Solution Approach 2:
The system implements partial action by collecting and transmitting only the subset of metrics that consumers have subscribed to, rather than collecting all possible telemetry data. This selective collection approach reduces network bandwidth consumption while ensuring that consumers receive complete data for their subscribed metrics, avoiding the excessive action of transmitting unnecessary data.
Data Source
AI summary
A telemetry processing system in a cluster network generates telemetry data from a plurality of telemetry producers and formats it into a structured format for storage in a datastore. Producers can define new metrics for use in a running network. Schema for a new metric is validated by a schema validator and stored in a telemetry catalog. Telemetry datasets for the new metric are sent from the producer for storage in a datastore. Consumers are notified as to the availability of the new metric, and those who elect to receive the new metric are sent the new datasets along with their previous datasets.