Parallel Processing Data Stream Normalization and Resampling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In parallel processing systems, managing and visualizing data from multiple streams across numerous processors or computers is challenging due to the large volume of data streams and the need for effective analysis and comparison of metrics over time, which existing methods fail to address efficiently.
Innovation Solution
A method that involves normalizing start and end times using a tree structure to resample data at specified intervals, calculating statistical derivatives, and displaying these derivatives as functions of time, allowing for efficient aggregation and comparison of multiple metrics on a common timeline, thereby reducing data transfer and processing bottlenecks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If data from multiple streams is aggregated and visualized using traditional methods (mean, variance, individual graphs), then comprehensive coverage of all data streams is achieved, but the complexity of data processing and visualization increases significantly
Solution Approach 1:
The patent segments the large volume of data streams by grouping them according to their start and end times. Data streams are divided into temporal segments or bins, allowing systematic processing of manageable portions rather than overwhelming all streams simultaneously. This segmentation reduces visualization complexity while maintaining comprehensive coverage.
Solution Approach 2:
The patent introduces a temporal dimension by organizing data streams according to their time intervals. Instead of processing all streams in a single flat space, data are distributed across temporal dimensions, creating a multi-dimensional structure that simplifies aggregation and visualization while preserving information from all streams.
2Measurement precision
If data streams are processed and visualized in detail, then accurate representation of system behavior is achieved, but data transfer and processing bottlenecks increase
Solution Approach 1:
The patent extracts only the essential temporal characteristics (start time, end time) from each data stream for the purpose of grouping. By taking out just the time interval information needed for organization, the system achieves accurate temporal representation without transferring or processing the entire detailed content of each stream, thus improving efficiency.
Solution Approach 2:
The patent performs preliminary grouping of data streams by their time intervals before detailed processing. By pre-organizing streams into temporal buckets, the system prepares data in advance for efficient aggregation and analysis, reducing the computational burden during actual processing and visualization phases.
3Productivity
If traditional aggregation methods are used to visualize multiple data streams, then overall statistics are obtained, but important details and variations between individual streams are lost
Solution Approach 1:
The patent segments data streams into temporal groups while preserving the ability to analyze both aggregated statistics and individual stream characteristics within each segment. This segmentation allows efficient overall analysis while maintaining access to detailed variations when needed, preventing information loss.
Solution Approach 2:
The patent applies different levels of analysis to different data streams based on their temporal characteristics. By organizing streams into time-based groups, the system can apply appropriate aggregation levels locally to each group while maintaining the ability to examine individual stream details when necessary, thus preserving important variations.
Data Source
AI summary
A method of analysing streams of metric data from a plurality of data processing sources (2) in a parallel processing system (1), using a computer (6). Each stream includes time stamped data associated with the respective data processing source in respect of a given metric which is sampled at intervals. For each stream of data, a start time and an end time are identified. A normalized start time and a normalized end time are determined across all streams. Sampling points are specified between the normalized start time and the normalized end time. For each stream of data, the data is re-sampled at the specified sampling points. For each sampling point, the re-sampled data across all the streams of data is processed to determine a statistical derivative of the data. A report is displayed which represents the values of the statistical derivative as a function of time. The process may be applied to various different metrics, and the results displayed at the same time, on a common time axis.