Incremental Kurtosis Calculation for Streamed Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for processing and analyzing Big Data, particularly streamed data, are inefficient and costly, as they require accessing and recalculating large datasets for kurtosis calculations, leading to slow response times and significant resource waste.
Innovation Solution
A computing system that incrementally calculates kurtosis by maintaining a computation subset size counter and updating components based on previous calculations, allowing for direct and indirect incremental calculations without revisiting all data elements, thus reducing computational load and resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional methods are used to calculate kurtosis on Big Data or streamed data, then calculation accuracy is maintained, but computational resources are significantly wasted and response time is slow
Solution Approach 1:
The system performs preliminary calculations of summation components (sum of values, sum of squares, sum of cubes, sum of fourth powers) during the data streaming process and stores them in memory. When kurtosis is needed, these pre-computed components are immediately available, eliminating the need to re-process the entire dataset from scratch.
Solution Approach 2:
The invention extracts only the necessary summary statistics (four power moments) from the large dataset and stores them separately. When kurtosis calculation is required, only these extracted components are accessed and processed, rather than accessing all individual data points in the computation subset.
2Measurement precision
If all data elements are accessed for kurtosis calculation, then accurate results are obtained, but processing time increases significantly
Solution Approach 1:
The system pre-computes and stores the summation components (sum of values, sum of squares, sum of cubes, sum of fourth powers) as data streams are received. When kurtosis is needed, these components are immediately available from memory, enabling fast calculation without re-scanning the entire dataset.
Solution Approach 2:
The invention introduces intermediate summary statistics (the four power moment components) as mediators between the raw data and the final kurtosis calculation. These intermediaries are pre-computed and stored, allowing the kurtosis to be calculated quickly using only these summary components rather than all individual data points.
3Measurement precision
If the computation subset size increases to include more data elements, then statistical accuracy improves, but computational complexity increases
Solution Approach 1:
The invention segments the kurtosis calculation into independent components (sum of values, sum of squares, sum of cubes, sum of fourth powers). Each component can be calculated and stored separately as data streams in, and when needed, these segmented components are combined to compute the final kurtosis. This segmentation allows the system to handle large computation subsets efficiently without increasing operational complexity.
Data Source
AI summary
The present invention extends to methods, systems, and computing system program products for incrementally calculating kurtosis for Big Data or streamed data in real time by incrementally calculating one or more components of kurtosis. Embodiments of the invention include incrementally calculating one or more components of a kurtosis for a modified computation subset based on the one or more components of the kurtosis calculated for a pre-modified computation subset and then calculating the kurtosis based on the incrementally calculated components. Incrementally calculating kurtosis avoids visiting all data elements in the modified computation subset and performing redundant computations thereby increasing calculation efficiency, saving computing resources and reducing computing system's power consumption.