Incremental Statistical Measure Calculation Method
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for estimating the mean, standard deviation, and variance of a dataset are inefficient, requiring large data storage, multiple passes through the dataset, and are prone to data overflow and slow adaptation to changes, especially when dealing with complex signals or sudden statistical changes.
Innovation Solution
A method that iteratively calculates statistical measures by initializing a counter and updating values based on each sample received, with conditions to freeze or reset the counter, allowing for real-time estimation without storing the entire dataset and avoiding data overflow, and is compatible with both real and complex-valued samples.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If classical methods are used to calculate statistical measures, then accuracy of mean, standard deviation and variance estimation is improved, but data storage requirements increase and processing time increases due to multiple passes over the dataset
Solution Approach 1:
The patent segments the statistical calculation process into incremental updates. Instead of processing the entire dataset at once, it calculates statistics by processing data in small sequential batches, updating running totals and means incrementally. This allows accurate statistical measures to be computed with minimal data storage, as only current batch data and running statistics need to be stored rather than the entire dataset.
Solution Approach 2:
The patent performs preliminary initialization by setting initial values for statistical measures before processing the main data stream. It establishes initial running totals, means, and other statistical parameters in advance, which then serve as the basis for incremental updates. This preliminary setup enables accurate statistics to be maintained without requiring storage of all historical data.
2Measurement precision
If classical methods are used to calculate variance, then measurement precision is improved, but processing time increases due to requiring at least two passes over the whole dataset
Solution Approach 1:
The patent implements continuous calculation of statistical measures by processing data in a single continuous pass. Instead of requiring separate passes for mean calculation and variance calculation, it continuously updates both statistics simultaneously as data flows through the system. The variance calculation uses the formula that incorporates the running mean and running total of squared deviations, allowing both statistics to be computed in one continuous traversal of the data stream.
3Adaptability or versatility
If sliding window or FIR filter methods are used to track mean, then adaptability to dynamic changes is improved, but processing time increases due to convolution requirements and window size determination complexity
Solution Approach 1:
The patent implements dynamic adaptation by continuously updating statistical measures as new data arrives, without requiring fixed window sizes or complex convolution operations. The running mean and variance are updated incrementally using new data points and previous statistics, allowing the system to adapt automatically to changing data distributions. This dynamic approach eliminates the need for predetermined window sizes and complex filtering algorithms.
4Productivity
If Welford's method is used to calculate statistical measures iteratively, then processing speed is improved, but data overflow issues occur when calculating variance and standard deviation
Solution Approach 1:
The patent changes the mathematical parameters used in the calculations to prevent overflow. Instead of accumulating raw sums of squared values that can overflow, it uses normalized formulas that divide by the number of samples processed so far. The variance calculation uses the formula: variance = (sum of squared deviations) / (n-1), where the sum is normalized by the sample count. This parameter transformation maintains numerical stability while preserving calculation speed.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method (200) of determining a statistical measure of a dataset, the method comprising, at a processor: receiving an initial sample of the dataset and initialising a counter value to zero (201); setting an initial value of the statistical measure based on the initial sample value (202); storing, in a memory, the initial value of the statistical measure (203); receiving further samples of the dataset sequentially (204), wherein for each successive sample received the method comprises: incrementing the counter value (205); determining an updated value of the statistical measure based on the previous, stored, value of the statistical measure, the current sample value, and the current counter value (206); updating the stored value of the statistical measure in the memory with the updated value of the statistical measure (207); and determining whether a reset condition is met (208), wherein in response to determining that the reset condition is met, reverting to step 201.