Virtual Stream Aggregation for Dynamic Sensor Scaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing stream processing systems are not scalable, either scaled up or scaled down, and fail to detect new streams or sensors coming online, or existing sensors going offline, and do not aggregate streams matching a client query into a single virtual stream that can be easily scaled-in or scaled-out on demand.
Innovation Solution
A streaming service generates a virtual stream by aggregating matching data streams, monitors changes in flow rate, and suggests adding or removing consumer processes to client devices, allowing for dynamic scaling and management of stream processing resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If separate stream processing is used for each sensor, then each sensor can be processed independently, but the system cannot scale up or down in an aggregate fashion and cannot detect new streams or sensors coming online
Solution Approach 1:
The patent merges multiple separate sensor streams into a single aggregated stream that represents all sensors. This allows the system to treat sensor data as a unified entity that can be scaled in or out together, enabling aggregate scaling while maintaining individual sensor processing capabilities through the unified stream structure.
Solution Approach 2:
The aggregated stream serves multiple functions simultaneously: it represents all sensors collectively, enables uniform scaling operations, provides a single interface for detecting new sensors, and maintains individual sensor data integrity. This multi-functionality resolves the contradiction by making the stream structure adaptable to various operations.
2Reliability
If separate stream processing is used for each sensor, then processing can be done independently, but the system does not detect new streams coming online that a client device should monitor
Solution Approach 1:
The system establishes a subscription mechanism where client devices subscribe to the aggregated stream in advance. When new sensors come online, the system automatically adds their streams to the aggregated stream, and subscribed clients receive notifications without needing to individually register for each new sensor. This preliminary subscription setup enables reliable detection of new streams while simplifying management.
3Reliability
If separate stream processing is used for each sensor, then individual processing is maintained, but existing sensors going offline are not automatically detected for stream termination
Solution Approach 1:
The system implements a feedback mechanism where sensor status changes (online/offline) are automatically detected and communicated to the stream management system. When sensors go offline, the system receives feedback about their status and automatically terminates their streams from the aggregated stream, eliminating the need for manual detection and termination management.
4Productivity
If aggregate sensor data suddenly increases due to an emergency, then all sensors are affected, but each sensor monitor process must separately decide to increase consumer processes
Solution Approach 1:
The patent combines all sensor streams into a single aggregated stream that represents the total sensor data load. When sensor data increases due to an emergency, the system can uniformly scale consumer processes across the aggregated stream rather than individually managing each sensor's consumer processes. This merging approach maintains processing throughput while significantly reducing the complexity of managing multiple consumer processes.
Data Source
AI summary
Systems and methods are disclosed for a streaming service generating an aggregation of data streams in response to a query from a client device for data streams matching the query. The aggregation of data streams matching the query is presented to the requesting client as a single virtual stream. The streaming service modifies the virtual stream in real-time by adding data streams matching the query that come online to the virtual stream, or deleting data streams from the virtual stream that go offline. The streaming service can partition the virtual stream if the data flow rate increases above a threshold, and reduce partitions if the data flow rate falls below a threshold. The client device can add or terminate consumer processes in response to changes in the data flow rate and partitioning.