Aggregate Data Reconstruction via Cluster Interpolation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing large amounts of network data is challenging due to storage requirements, complexity in organizing and processing, and the impact on network traffic, making it inefficient to store and retrieve all collected data.
Innovation Solution
The method involves partitioning collected raw data into clusters, determining average points, and storing aggregate data, which allows for the reconstruction of data points through interpolation, reducing storage needs and improving processing efficiency by using aggregate data to approximate original data values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all collected network performance data is stored, then data availability and completeness are improved, but storage space requirements increase significantly
Solution Approach 1:
The patent extracts and stores only aggregate data (sums, counts, averages) from the original network performance data, separating the essential statistical information from the detailed raw data. This allows the system to maintain data availability for analysis while dramatically reducing storage requirements by keeping only the aggregated results rather than all individual data points.
Solution Approach 2:
The patent transforms the data representation by changing from storing individual data points to storing aggregate parameters (sums, counts, averages). This parameter transformation enables the system to preserve the essential characteristics of the data for analysis purposes while occupying minimal storage space, as aggregate parameters compact large volumes of data into concise statistical representations.
2Ease of operation
If all collected data is indexed and organized for future use, then data retrieval capability is improved, but processing complexity and time increase
Solution Approach 1:
The patent extracts only the essential aggregate parameters from the data and stores them in a simplified structure. By removing the complexity of organizing and indexing every individual data point and keeping only the aggregated results, the system maintains efficient data retrieval capability while significantly reducing the processing complexity and time required for data organization.
3Quantity of substance
If large databases are implemented as distributed databases with complex indexing schemes, then data storage capacity is improved, but query execution time increases
Solution Approach 1:
The patent creates a compact copy of the data in aggregate form (sums, counts, averages) that can be stored in a simple, centralized structure. This aggregate copy serves as a simplified representation that can be queried and processed much faster than the original distributed database, trading detailed data access for rapid execution of aggregate queries while maintaining the storage capacity of the original system.
4Reliability
If large amounts of data are transmitted between nodes for collection and processing, then data completeness is improved, but network bandwidth consumption increases
Solution Approach 1:
The patent merges multiple data points into single aggregate values (sums, counts, averages) at the source nodes before transmission. By combining numerous individual data points into compact aggregate representations, the system maintains data completeness for analysis purposes while dramatically reducing the volume of data transmitted across the network, thus conserving bandwidth.
Data Source
AI summary
In an embodiment, a method comprises dividing collected data into data clusters based on proximity of the data and adjusting the clusters based on density of data in individual clusters. Based on first data points in a first cluster, a first average point in the first cluster is determined. Based on second data points in a second cluster, a second average point in the second cluster is determined. Aggregate data, comprising the first average point and the second average point, are stored in storage. Upon receiving a request to provide data for a particular coordinate, the reconstructed data point is determined by interpolating between the first average point and the second average point at the particular coordinate. Accordingly, aggregated data may be stored and when a request specifies data that was not actually stored, a reconstructed data point with an approximated data value may be provided as a substitute.