Distributed Aggregation Map for Large Data Volumes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data aggregation methods face limitations when dealing with large volumes or sparse data, as they require impractical aggregation maps that can exceed memory capacity and degrade query performance.
Innovation Solution
A distributed map is partitioned across multiple devices, allowing efficient aggregation from one dimension level to another, using either in-memory or disk-based implementations, and can be periodically recreated to include new data, utilizing distributed hash tables and batch processing to manage large volumes of data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a conventional aggregation map is used to map large volumes of data, then data aggregation can be performed, but the aggregation map becomes unwieldy and exceeds memory capacity
Solution Approach 1:
The aggregation map is divided into multiple partitions distributed across different devices in a network. Each device stores a subset of the aggregation map, allowing the system to handle large volumes of data without requiring a single device to hold the entire map in memory. This segmentation resolves the contradiction by distributing the burden across multiple devices.
Solution Approach 2:
The solution transitions from a single-device memory structure to a multi-device distributed network structure, adding the dimension of spatial distribution across network devices. This allows the aggregation map to scale beyond the memory capacity of any single device while maintaining efficient data aggregation capabilities.
2Quantity of substance
If a conventional aggregation map is used for sparse data, then data aggregation can be performed, but query performance degrades
Solution Approach 1:
The distributed aggregation map partitions data across multiple devices, allowing sparse data to be organized efficiently in a distributed structure. This segmentation enables faster query performance by reducing the search space on any single device and allowing parallel access across the network.
3Quantity of substance
If the aggregation map is distributed across multiple devices, then machine-driven limitations on map size are removed, but system complexity increases
Solution Approach 1:
The distributed aggregation map system provides universal functionality across multiple devices, allowing any device in the network to contribute to and access the aggregation map. This multi-functionality approach standardizes operations across devices, managing complexity through consistent interfaces and protocols.
Solution Approach 2:
The system introduces intermediary components such as distributed hash tables and network communication protocols that mediate between individual devices and the overall aggregation map. These intermediaries abstract the complexity of distributed storage and retrieval, allowing devices to interact with the aggregation map through standardized interfaces.
Data Source
AI summary
Processes, systems and computer program products for data management. A distributed map is provided that maps data from a first dimension level in a hierarchy to a second dimension level in the hierarchy. The data is aggregated from the first dimension level to the second dimension level based on the mapping. The distributed map is partitioned across multiple devices.