WebSocket Data Ingestion for Analytics Platforms
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Gathering time series data from diverse sources into an analytics data warehouse is challenging due to the high cost and maintenance requirements of bespoke collectors and the 'funneling effect' caused by aggregating data from multiple IT infrastructure components, which restricts parallelism and leads to inefficiencies in data transfer through REST APIs.
Innovation Solution
Utilizing WebSockets to enable asynchronous, bi-directional communication for efficient data ingestion, leveraging the asynchronous nature of WebSockets in combination with REST APIs to allow parallel data streams from managed elements to a data analytics platform, thereby reducing latency and enhancing scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If bespoke collectors are built for each element manager using data export protocols, then good runtime performance is achieved, but the cost and maintenance requirements increase significantly as the number of data sources increases
Solution Approach 1:
The patent implements a universal collector that can ingest data from multiple element managers through a single standardized interface (WebSocket). Instead of building separate bespoke collectors for each data source, the system uses one multi-functional collector that adapts to different element managers, thereby reducing the number of components and maintenance overhead while maintaining efficient data collection
2Ease of operation
If element managers aggregate data from multiple IT infrastructure components, then data collection is simplified, but the inherent parallelism is masked and scale-out capability is prevented due to the funneling effect
Solution Approach 1:
The patent segments the data collection architecture by allowing each managed element to have its own dedicated WebSocket connection to the collector. This segmentation prevents the funneling effect where all data must pass through a single aggregation point, enabling parallel data streams from multiple sources to be processed simultaneously, thus maintaining both simplicity and scalability
3Adaptability or versatility
If REST APIs are used for data transfer, then standard communication protocols are utilized, but latency increases and real-time performance is reduced
Solution Approach 1:
The patent changes the communication parameter from REST API (request-response model with higher latency) to WebSocket (persistent connection with push capability). This parameter change maintains adaptability by using a standardized protocol while dramatically reducing latency and enabling real-time data transfer, as the persistent connection allows data to be pushed immediately without repeated connection establishment overhead
Data Source
AI summary
Example embodiments of the present invention relate to a method, a system, and a computer program product for data analytics. The method includes receiving a plurality of first data streams from respective managed elements in a network and partitioning data of the plurality of first data streams according to an attribute regarding the data into a partitioned second data stream. The partitioned second data stream then may be streamed toward a data analytics platform for consumption by the data analytics platform. In a preferred embodiment, WebSockets are used.