Iterative Variance Calculation for Big Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for processing and analyzing Big Data are inefficient, leading to redundant data access and computation, especially in real-time decision-making scenarios, due to the large volume and rapid growth of data, which results in slow response times and significant resource waste.
Innovation Solution
The system iteratively calculates variance and standard deviation by modifying a computation subset by adding or removing data elements, using pre-calculated components to directly or indirectly update the calculations, thereby avoiding redundant computations and reducing the need to access all data elements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional methods recalculate variance and standard deviation by accessing all data elements in the computation subset, then calculation accuracy is maintained, but calculation time increases and computing resources are wasted
Solution Approach 1:
The computation subset is segmented into a stored portion (in memory) and a distributed portion (in cloud storage). The patent divides data elements into two groups: those frequently accessed kept in memory, and those less frequently accessed stored in cloud storage. This segmentation allows the system to work with only the necessary subset of data elements during recalculation, avoiding the need to access all n data elements while maintaining calculation accuracy.
Solution Approach 2:
The system performs preliminary actions by pre-calculating and storing intermediate results (such as sums, means, and partial variance components) in memory before they are needed for recalculation. When data changes occur, the system uses these pre-computed values to quickly update the variance and standard deviation without重新计算 all data elements, thus reducing calculation time while preserving accuracy.
2Reliability
If all data elements in the computation subset are accessed for recalculation, then complete and accurate results are obtained, but redundant data access occurs and computing resources are significantly wasted
Solution Approach 1:
The patent extracts only the necessary data elements from the computation subset for recalculation. Instead of accessing all n data elements, the system identifies and extracts only those elements that have changed or are relevant to the current calculation needs. This extraction principle allows the system to maintain result completeness by including all necessary data while excluding redundant elements, thereby reducing computing resource waste.
Solution Approach 2:
The system applies local quality by storing different portions of data elements with different qualities (access frequency, recency, importance) in different locations. Frequently accessed data elements are stored in memory with high availability, while less frequently accessed elements are stored in cloud storage. This local differentiation allows the system to access only the necessary high-quality data elements for recalculation, ensuring result completeness while minimizing energy consumption.
3Measurement precision
If the computation subset size n is increased to include more data elements, then statistical accuracy improves, but the time and resources required for recalculation increase significantly
Solution Approach 1:
The patent implements a nested structure where the computation subset is nested within the larger data set, and further nested into two levels: memory-stored portion and cloud-stored portion. This nested organization allows the system to efficiently manage large computation subsets by providing hierarchical access - frequently needed data is accessible at the innermost level (memory), while less frequently needed data is at outer levels (cloud storage). This enables statistical accuracy to be maintained with large n while improving recalculation speed through selective access.
Solution Approach 2:
The system adds a spatial dimension to data storage by distributing data elements across multiple dimensions: memory storage and cloud storage. This dimensional organization allows the computation subset to be virtually unlimited in size while maintaining fast access to critical data elements. The patent leverages this dimensional structure to include more data elements in the computation subset for improved statistical accuracy, while the distributed architecture ensures that recalculation can proceed efficiently by accessing only the necessary portions.
Data Source
AI summary
The present invention extends to methods, systems, and computing system program products for iteratively calculating variance and/or standard deviation for Big Data. Embodiments of the invention include iteratively calculating one or more components of a variance and/or a standard deviation in a modified computation subset based on iteratively calculated one or more components of the variance and/or the standard deviation calculated for a previous computation subset and then calculating the variance and/or the standard deviation based on the iteratively calculated components. Iteratively calculating the components of variance and/or standard deviation avoids visiting all data elements in the modified computation subset and performing redundant computations thereby increasing calculation efficiency, saving computing resources and reducing computing system's power consumption.