Iterative Kurtosis Calculation for Big Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for processing and analyzing Big Data are inefficient and resource-intensive, particularly when calculating statistical metrics like kurtosis, due to redundant data access and computation, which leads to slow response times and significant resource waste.
Innovation Solution
The system iteratively calculates kurtosis by modifying a computation subset by removing and adding data elements, using pre-calculated components to directly or indirectly update the kurtosis components, thereby avoiding redundant computations and reducing the need to access all data elements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all data elements in the computation subset are accessed to recalculate kurtosis after data changes, then calculation accuracy is maintained, but computing time and resource consumption increase significantly
Solution Approach 1:
The computation subset is segmented into two parts: data elements that remain unchanged and data elements that change (added or removed). Only the changed elements are processed in each iteration, while unchanged elements are excluded from repeated access. This segmentation allows the system to maintain calculation accuracy by focusing only on relevant data changes rather than reprocessing the entire subset.
Solution Approach 2:
The system performs preliminary identification of changed data elements before initiating recalculation. By detecting which elements have been added or removed from the computation subset, the system can prepare targeted update operations that only affect the necessary portions of the kurtosis calculation, avoiding redundant processing of unchanged elements.
2Reliability
If all data elements are repeatedly accessed for recalculation, then complete and accurate results are obtained, but computing resources are wasted
Solution Approach 1:
The system extracts and processes only the changed data elements (added or removed elements) from the computation subset, separating them from the unchanged elements. By taking out only the necessary components for recalculation, the system maintains calculation completeness while eliminating the energy waste associated with reprocessing unchanged data elements.
Solution Approach 2:
Instead of performing complete recalculation on all data elements (excessive action), the system performs partial recalculation focused only on the changed elements. This partial action is sufficient to maintain calculation reliability while significantly reducing computing resource consumption compared to full recalculation.
3Measurement precision
If the computation subset size is increased to improve statistical accuracy, then measurement precision improves, but the complexity of processing and power consumption increase
Solution Approach 1:
The large computation subset is segmented into unchanged and changed elements. This segmentation reduces processing complexity by allowing the system to handle only the small portion of changed elements rather than the entire large subset, while still maintaining statistical accuracy benefits from the large overall subset size.
Solution Approach 2:
The system implements dynamic processing that adapts to data changes. Rather than static full recalculation, the dynamic approach identifies and processes only the changed elements, reducing processing complexity while maintaining the statistical accuracy benefits of the large computation subset.
Data Source
AI summary
The present invention extends to methods, systems, and computing system program products for iteratively calculating kurtosis for Big Data. Embodiments of the invention include iteratively calculating one or more components of a kurtosis in a modified computation subset based on the one or more components of the kurtosis calculated for a previous computation subset and then calculating the kurtosis based on the iteratively calculated components. Iteratively calculating kurtosis avoids visiting all data elements in the modified computation subset and performing redundant computations thereby increasing calculation efficiency, saving computing resources and reducing computing system's power consumption.