Multidimensional Data Partitioning for High-Volume Event Aggregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current event management systems face inefficiencies in handling high volumes of data from web applications and IoT devices, leading to resource-intensive processing and latency issues, especially in multi-tenant cloud computing environments where real-time tracking of metrics is desired.
Innovation Solution
Implementing a data aggregation manager that utilizes multidimensional partitioning and data deduplication to efficiently process and store data streams by partitioning data based on source organization, asset identifier, and target audience, reducing redundant data and improving scalability and throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If current event management systems process high volumes of data from web applications and IoT devices, then data collection capability is improved, but resource consumption increases and latency occurs
Solution Approach 1:
The patent applies segmentation by dividing the data processing system into multiple worker nodes that independently process different partitions of event data. Each worker node handles a specific subset of data streams, allowing parallel processing and reducing the resource burden on any single node. This segmentation enables the system to scale horizontally by adding more worker nodes to handle increasing data volumes without proportionally increasing resource consumption per node.
2Speed
If data is processed in real-time to provide real-time user experience, then responsiveness is improved, but resource intensity increases significantly
Solution Approach 1:
The patent implements preliminary action by pre-partitioning event data into distinct data streams based on event type, source, and other characteristics before processing. This pre-organization of data allows worker nodes to efficiently process specific partitions without the overhead of dynamic data routing and filtering during real-time processing. The preliminary partitioning reduces computational overhead and enables faster, more resource-efficient real-time processing.
3Adaptability or versatility
If data is distributed across multiple nodes for scalability, then system capacity is improved, but data organization and retrieval complexity increases
Solution Approach 1:
The patent applies local quality by assigning specific data partitions to specific worker nodes based on deterministic partitioning keys. Each worker node maintains local knowledge of its assigned partitions and can independently process and retrieve data from its local partition without requiring global coordination. This localizes data management operations, reducing the complexity of distributed data organization while maintaining scalability through consistent hashing or range-based partitioning strategies.
Data Source
AI summary
A method enables data aggregation in a multi-tenant system. The method includes receiving, at a data aggregation manager, an event from an activity tracking component, generating, by the data aggregation manager, a first hash value based on a first identifier in the event, generating, by the data aggregation manager, a second hash value based on a second identifier in the event, and storing event message information to an entry of a metric storage database in a partition according the first hash value and the second hash value.