Distributed Aggregation on Overlay Network Tiers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed monitoring systems, a single monitoring computer system often becomes overwhelmed when handling large volumes of data from multiple sources, as its resources are limited by hardware constraints, making it impractical to continue upgrading indefinitely.
Innovation Solution
Implementing a distributed aggregation method on an overlay network with multiple tiers of nodes, where pre-aggregated data is routed across nodes, allowing for scalable data processing and handling increased volumes without relying solely on a central node's capacity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single monitoring computer system is used to aggregate input data streams from multiple sources, then the system can maintain centralized control and simplified architecture, but the system becomes overwhelmed when the volume of input data exceeds the hardware capacity of the monitoring computer
Solution Approach 1:
The patent divides the aggregation system into multiple tier levels (first tier nodes, second tier nodes, and central node) where each node handles a portion of the data aggregation workload. First tier nodes perform initial aggregation of event streams from multiple sources, second tier nodes aggregate results from first tier nodes, and the central node performs final aggregation. This segmentation distributes the processing load across multiple computers, enabling the system to handle larger data volumes without overwhelming a single monitoring computer.
2Productivity
If the hardware resources of the monitoring computer system are increased to handle larger data volumes, then the data aggregation capacity can be improved, but it becomes economically infeasible and technically impractical beyond certain limits due to hardware constraints
Solution Approach 1:
Instead of continuously upgrading a single monitoring computer's hardware, the patent segments the aggregation function across multiple computers at different tier levels. This allows the system to scale horizontally by adding more nodes rather than vertically upgrading expensive hardware, making it economically feasible to handle growing data volumes.
Solution Approach 2:
The patent transitions from a single-dimension scaling approach (upgrading one computer's hardware) to a multi-dimensional approach by introducing tier levels and distributing nodes across the network. This dimensional change enables scaling through network topology expansion rather than hardware capacity expansion, bypassing hardware limits.
3Productivity
If a distributed aggregation system with multiple tiers is implemented, then the data processing capacity and scalability are improved, but the system complexity and routing overhead increase
Solution Approach 1:
The patent organizes the distributed system into clearly defined tier levels with specific responsibilities at each level, which segments the complexity into manageable portions. First tier nodes handle event stream aggregation, second tier nodes handle intermediate aggregation, and the central node handles final aggregation. This segmentation makes the overall system complexity more manageable compared to a fully distributed peer-to-peer approach.
Solution Approach 2:
The patent introduces intermediary nodes (second tier nodes) that mediate between first tier nodes and the central node. These intermediary nodes simplify routing by providing an intermediate aggregation layer, reducing the direct communication burden on the central node and organizing data flow through structured intermediate steps rather than direct point-to-point routing.
Data Source
AI summary
The present invention extends to methods, systems, and computer program products for distributed aggregation on an overlay network. Embodiments of the invention utilize tiers of nodes that are cascaded in a layered system. Each tier reduces the size of data by orders of magnitude through pre-aggregation. Thus, high volume streams of messages can be reduced to lower volume streams at large scales, such as, for example, the Internet. No central coordination is used; thus there is no central point of failure or bottleneck. When a node fails, other nodes in the same tier as the failing node automatically take over the responsibilities of the failed node.