Stream Analytics Processing via Embedded Data Sketches
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern communications networks face challenges in efficiently processing vast amounts of data from diverse sources, leading to high sampling latencies and substantial compute resource requirements for monitoring and maintaining quality of service, due to the complexity of configuring data sketches to sample relevant information from data lakes.
Innovation Solution
A real-time analytics system (RETA) is implemented, embedding data sketches in load balancers and probes to directly process protocol data units and records, generating key performance indicators (KPIs) in real-time, with the ability to aggregate values from different streams and provide operating profiles for network entities, enabling real-time monitoring and anomaly detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data sketches scan and sample data in data lakes to generate KPI values, then network monitoring and quality of service maintenance are enabled, but sampling latencies increase and substantial compute resources are required
Solution Approach 1:
The patent applies preliminary action by pre-configuring data sketches within network elements (load balancers, probes, network functions) to process data streams in real-time as they pass through the network. This eliminates the need to later scan and sample data from data lakes, as the KPI generation happens proactively at the source, thereby reducing sampling latency while maintaining monitoring reliability
2Reliability
If data sketches scan and sample data in data lakes to generate KPI values, then network monitoring and quality of service maintenance are enabled, but substantial compute resources are required
Solution Approach 1:
The patent extracts the data sketch processing functionality directly from the centralized data lake processing model and embeds it within distributed network elements (load balancers, probes, network functions). This distributes the compute workload across multiple network elements, eliminating the need for substantial centralized compute resources to scan and sample data lakes, while maintaining reliable network monitoring capability
3Adaptability or versatility
If data in data lakes is stored in different structured and unstructured formats from different data sources, then comprehensive network data collection is achieved, but configuring sketches to sample relevant information becomes complex
Solution Approach 1:
The patent implements universality by designing data sketches with standardized interfaces and configuration mechanisms that can process multiple data formats (structured and unstructured) from different sources. The sketches are configured with universal templates that automatically adapt to various PDU types and data formats, thereby maintaining comprehensive data collection capability while reducing sketch configuration complexity
Data Source
AI summary
A communications network, comprising: an array of load balancers and probes for monitoring operations of the network; and at least one data sketch embedded in a load balancer and/or a probe of the array that is operable to process data comprised in protocol data units (PDUs) that are streamed to the array.