Balanced Tree Forest Partitioning for Memory-Constrained Data Aggregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computer systems processing data streams face inefficiencies and unreliability due to limited computational resources when handling large amounts of data from multiple sources, particularly in distributed storage systems using tree-based structures, which require significant memory and lead to increased costs when scaling up.
Innovation Solution
A computer system and storage system architecture that stores only a part of the forest of balanced trees in memory, with the remaining nodes stored in a connected storage system, allowing for efficient and reliable data aggregation in sliding windows by minimizing memory usage and enabling processing of larger data sets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If all nodes of the forest of balanced trees are stored in memory, then data aggregation can be performed efficiently, but memory requirements become excessively large
Solution Approach 1:
The patent segments the forest of balanced trees into multiple partitions, where only the current partition is loaded into memory while other partitions are stored in external storage. This allows efficient processing of the active partition while minimizing memory usage, and enables handling of datasets larger than available memory through partitioned storage
2Reliability
If computer systems are provided with more powerful computational resources to process data from multiple data streams, then processing reliability improves, but system cost increases
Solution Approach 1:
The patent divides the forest into multiple partitions that can be processed independently. This segmentation allows the system to handle multiple data streams by distributing them across different partitions, improving processing reliability through parallel processing while avoiding the need for exponentially more powerful hardware resources
Solution Approach 2:
The patent introduces a new dimension of partitioning and parallel processing, transforming the problem from requiring more powerful single-system resources to using multiple simpler systems that can process different partitions simultaneously, thereby improving reliability without proportionally increasing cost
Data Source
Figure 1a
Figure 1b
Figure 1c
AI summary
Computer systems are provided for distributed storage of data structured as forests of balanced trees of nodes, each node including data-elements and the forests comprising levels. The nodes comprise first end nodes at first side of the forest, second end nodes at second side of the forest, and intermediate nodes between first and second end nodes. The computer systems comprise memories to store at least the first and second end nodes; connectors for implementing connections with storage systems storing intermediate nodes, so that exchange of nodes with the storage systems is performed through said connections; and processors to update the nodes stored in the memories according to updating criteria, and to exchange nodes with the storage systems through the connections according to exchange criteria. Storage systems are also provided, along with methods and computer programs that are performable by the computer systems.