Analytics Collection Scaling in Software-Defined Networking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In network telemetry, consumers of analytics data often become bottlenecks due to overwhelming event processing, leading to lost events from resources, as their processing capacity is overwhelmed by the rate and volume of analytics data received.
Innovation Solution
A system for analytics collection in software-defined networking that includes an analytics engine and a resource interface, which employs load balancing, fine-grained filtering, and event distribution algorithms to manage and process telemetry data efficiently, ensuring that only relevant events are processed and delivered to consumers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If consumers process all analytics data from resources, then complete data processing is achieved, but consumer processing capacity becomes overwhelmed and events are lost
Solution Approach 1:
The patent segments the consumer architecture into multiple consumer instances (consumer group) that collectively process events from resources. Each consumer handles a subset of the event stream, dividing the processing load to prevent any single consumer from becoming overwhelmed while maintaining complete event processing across the consumer group.
Solution Approach 2:
The patent introduces an event broker as an intermediary component between resources and consumers. The event broker receives events from resources, manages the consumer group, and distributes events to appropriate consumers using load balancing algorithms. This intermediary abstracts the complexity of load management from individual consumers and enables scalable event processing.
2Loss of information
If consumers process all incoming analytics events, then comprehensive analytics are generated, but IO from resources to consumers becomes overwhelmed
Solution Approach 1:
The patent implements filtering mechanisms that apply partial processing to event streams. Filters selectively pass only relevant events to consumers based on predefined criteria, reducing the volume of events that require full processing. This partial action approach maintains analytics completeness for relevant events while reducing overall IO bandwidth consumption.
Solution Approach 2:
The patent extracts and removes irrelevant events from the event stream through filtering mechanisms before events reach consumers. By taking out unnecessary events early in the processing pipeline, the system reduces IO bandwidth consumption while preserving all events that are relevant for analytics generation.
3Productivity
If multiple consumers subscribe to events from resources, then analytics processing capacity increases, but load distribution becomes unbalanced
Solution Approach 1:
The patent implements load balancing algorithms that use feedback from consumer performance metrics to dynamically distribute events. The event broker monitors consumer processing rates and adjusts event distribution in real-time, routing more events to underutilized consumers and fewer events to overloaded consumers, thereby maintaining balanced load distribution across the consumer group.
Solution Approach 2:
The patent creates a dynamic event distribution system where the event broker continuously adapts event routing based on current consumer states. Consumer groups can be dynamically added or removed, and event distribution algorithms adjust in real-time to changing load conditions, enabling flexible scaling of analytics processing capacity.
Data Source
AI summary
A system for analytics collection from distributed resource in a software-defined networking. The system comprises: an analytics engine; and a resource interface communicated with the analytics engine, the resource interface being configured to collect telemetry data from the resource and to transmit the telemetry data to the analytics engine.