Decremental Z-score Calculation for Big Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for processing Big Data and streamed data are inefficient, particularly in real-time decision-making scenarios, as they require extensive computational resources and are costly, and traditional Z-score calculations are time-consuming and resource-intensive due to the need to recalculate statistics upon data element removal.
Innovation Solution
A computing system that decrementally calculates Z-scores by maintaining a computation subset size counter and updating components based on previous calculations, avoiding the need to revisit all data elements and store the entire computation subset, thereby increasing efficiency and reducing resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional Z-score calculation methods are used on Big Data or streamed data, then calculation accuracy is maintained, but computational resources and time consumption increase significantly
Solution Approach 1:
The patent segments the Z-score calculation into independent components (mean, standard deviation, variance) that can be updated incrementally. Instead of recalculating all statistics from scratch, each component is updated separately based on the removed data element, enabling efficient decremental updates while maintaining accuracy.
Solution Approach 2:
The patent performs preliminary calculations by maintaining running statistics (mean, variance, count) as data elements are added to the computation subset. When an element is removed, these pre-maintained statistics enable rapid updates without revisiting all data elements, thus improving calculation efficiency.
2Reliability
If traditional Z-score calculation methods are used, then complete statistical analysis is achieved, but computing resources and power consumption increase
Solution Approach 1:
The patent extracts only the necessary statistical components (mean, variance, count) required for Z-score calculation and maintains them separately. This extraction approach allows updates to be performed using only the removed data element and the extracted components, avoiding the need to process or store the entire data set, thus reducing computing resource consumption.
3Measurement precision
If all data elements are revisited for recalculation after data removal, then calculation accuracy is maintained, but time consumption increases
Solution Approach 1:
The patent implements dynamic updating of statistical components where the mean, variance, and count are continuously adjusted as data elements are added or removed. This dynamic approach allows the statistics to adapt in real-time without static recalculation of all data elements, maintaining accuracy while reducing time consumption.
4Reliability
If the entire computation subset is stored for recalculation, then complete statistical analysis is possible, but memory usage and storage requirements increase
Solution Approach 1:
The patent extracts and maintains only the essential statistical summaries (mean, variance, count, and optionally standard deviation) needed for Z-score calculation. This extraction eliminates the need to store the entire computation subset, reducing memory usage and storage requirements while preserving the capability to perform complete statistical analysis through these summarized components.
Data Source
AI summary
The present invention extends to methods, systems, and computing system program products for decrementally calculating Z-score for Big Data or streamed data. Embodiments of the invention include decrementally calculating one or more components of a Z-score for a modified computation subset based on one or more components of a Z-score calculated for a pre-modified computation subset and then calculating a Z-score for a selected data element in the modified computation subset based on one or more of the decrementally calculated components. Decrementally calculating Z-score avoids visiting all data elements in the modified computation subset and performing redundant computations thereby increasing calculation efficiency, saving computing resources and reducing computing system's power consumption.