Streaming Data Correlation Computation Using Sliding Window Binning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing algorithms for computing nonparametric correlations like Spearman's rank and Kendall's tau are inefficient for real-time processing of streaming data due to high computational time and memory requirements, especially when dealing with large datasets and outliers, making them unsuitable for dynamic environments such as sensor networks and financial analysis.
Innovation Solution
A system that includes a processor and a non-transitory computer-readable medium with instructions to iteratively update correlations using bin data, allowing for real-time computation of nonparametric correlations by incrementing bin data values and computing correlation values between variables without the need to store all data points, using a sliding window mechanism to forget outdated data and focus on recent observations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If standard batch algorithms are used to compute nonparametric correlations, then measurement precision is maintained, but productivity deteriorates due to slow computation speed and high memory requirements
Solution Approach 1:
The patent segments the data stream into fixed-size windows and processes each window independently to compute correlations. This allows the system to handle large datasets by dividing them into manageable chunks, enabling real-time processing while maintaining correlation accuracy through systematic segmentation of the data flow.
Solution Approach 2:
The patent changes the computational parameters by using optimized algorithms that compute correlations with reduced time complexity. By adjusting the computational approach from traditional batch processing to streamlined streaming algorithms, the system achieves faster computation speeds while preserving measurement precision through parameter-optimized correlation calculations.
2Measurement precision
If all streaming data is stored in memory for processing, then measurement precision is improved, but device complexity increases due to excessive memory requirements
Solution Approach 1:
The patent extracts only the essential data elements needed for correlation computation from the full data stream, discarding redundant information. By extracting and retaining only the critical variables and their relationships, the system maintains correlation analysis accuracy while significantly reducing memory requirements and device complexity.
Solution Approach 2:
The patent implements dynamic memory management where the system adapts its memory usage based on the current data window and computational needs. By dynamically allocating and releasing memory resources as data streams through the system, the platform maintains high correlation precision without requiring excessive permanent memory capacity.
3Productivity
If traditional algorithms process streaming data in real-time, then productivity is improved, but measurement precision deteriorates due to inability to handle outliers and dynamic data effectively
Solution Approach 1:
The patent converts the presence of outliers and dynamic variations in the data stream from harmful factors into beneficial information for correlation analysis. By designing the algorithm to specifically account for and leverage these variations rather than smooth them away, the system maintains real-time processing speed while improving measurement precision through robust correlation calculations that adapt to data dynamics.
4Measurement precision
If batch processing is used for correlation computation, then measurement precision is maintained, but loss of time increases due to inability to provide real-time results
Solution Approach 1:
The patent implements continuous correlation computation that operates uninterrupted as data streams through the system. By maintaining continuous processing rather than periodic batch operations, the system eliminates computation delays and provides real-time correlation results while preserving measurement precision through ongoing analytical computation.
Data Source
AI summary
A data streaming environment provides a summary of streaming data from a sensor that is an Internet of things device. An input interface receives the streaming data. A processor is communicatively coupled to the input interface for processing the streaming data. The processed streaming data includes, but is not limited to, a plurality of records and variables that describe a characteristic of a physical object. A computer-readable medium has instructions stored thereon that, when executed by the processor, cause the processor to execute a correlation update application with the received streaming data to provide a correlation between two variables of the streaming data. The non-transitory computer-readable medium further stores sum and bin data for the correlation update application to compute the correlation. The output interface provides the processed streaming data to be visually presented in one or more data graphs on a display device.