Distributed Data Aggregation Service Using Map-Reduce and Hash Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The complexity of provisioning, administering, and managing large input data sets and associated physical computing resources in distributed computer networks has become increasingly challenging, particularly in data centers that require efficient data aggregation and storage solutions.
Innovation Solution
The implementation of a Distributed Data Aggregation (DDA) service that utilizes a map-reduce architecture and distributed storage structures, such as distributed hash tables, to manage multi-stage data manipulation operations across multiple computing nodes, enabling efficient data aggregation and storage of large data sets in a scalable and reliable manner.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data aggregation and storage operations are performed using traditional centralized methods, then the system is easier to manage, but the system cannot efficiently handle very large data sets and lacks scalability
Solution Approach 1:
The patent divides the data aggregation and storage system into multiple independent computing nodes distributed across a network. Each node processes portions of the data set independently, allowing the system to handle very large data sets by parallel processing. The data itself is segmented and distributed across multiple nodes, with each node storing and processing specific data partitions, thereby achieving scalability without proportionally increasing management complexity.
2Adaptability or versatility
If a distributed computing architecture is implemented to handle large data sets, then the system becomes more scalable, but the complexity of provisioning and managing computing resources increases
Solution Approach 1:
The patent implements self-service mechanisms where computing nodes automatically discover available resources on the network and autonomously join the distributed system. Nodes self-provision by detecting other nodes and automatically integrating themselves into the data aggregation workflow without requiring manual provisioning. This automatic self-configuration and self-integration capability allows the system to scale dynamically while minimizing the operational burden of resource management.
3Reliability
If data is stored in a centralized location, then data access is simpler, but the system loses reliability and availability when handling large distributed data sets
Solution Approach 1:
The patent segments the stored data into multiple partitions distributed across different computing nodes in the network. Each node maintains local storage for its assigned data partition, eliminating the single point of failure inherent in centralized storage. This distributed storage architecture ensures that data remains accessible even if individual nodes fail, thereby improving reliability and availability while maintaining manageable storage complexity through automated data distribution and retrieval mechanisms.
Data Source
AI summary
Techniques are described for managing aggregation of data in a distributed manner, such as for a particular client based on specified configuration information. The described techniques may include storing aggregated data values for an OLAP cube or other data structure in a distributed manner, such as in some situations in a distributed hash table. The aggregated data values to be stored may be generated in various manners, such as by performing multi-stage data manipulation operations—for example, a map-reduce architecture may be used, with a first stage involving the use of one or more specified map functions to be performed, and with at least a second stage involving the use of one or more specified reduce functions to be performed.