Block-Based Variance Computation Using One-Pass Mean Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing techniques for determining variance are inefficient, requiring multiple passes through the data set and hardware, especially when dealing with large datasets, and are not optimized for block-based hardware, leading to processing delays and hardware inefficiencies.
Innovation Solution
A block-based iterative one-pass processing method that leverages block-based hardware to process entire blocks of data simultaneously, using the Welford algorithm to iteratively update running mean and variance, combining blocks to achieve efficient variance computation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If multiple processors are used to process data in parallel, then processing speed is improved, but hardware complexity and cost increase
Solution Approach 1:
The patent divides the data set into multiple blocks that can be processed in parallel by a single hardware component. The variance computation is segmented into block-level operations (computing mean and sum of squares for each block) and then combined at the end, allowing one processor to handle what would traditionally require multiple processors.
Solution Approach 2:
The patent introduces a block dimension to the processing architecture, transforming the problem from processing individual data points to processing blocks of data. This dimensional change allows a single processor to achieve parallel processing capabilities by operating on multi-dimensional data blocks rather than scalar values.
2Productivity
If traditional processors are used for data processing, then hardware versatility is maintained, but processing efficiency decreases
Solution Approach 1:
The patent changes the operational parameters of the hardware by implementing specialized block-based processing logic that performs vector operations (computing mean and sum of squares for multiple data points simultaneously). This parameter change enables traditional processors to achieve higher efficiency by optimizing their operation mode for statistical computations.
Solution Approach 2:
The patent makes a single hardware component universal by enabling it to process multiple types of data blocks (different sizes, different data types) through the same block-based variance computation algorithm, maintaining hardware versatility while improving processing efficiency for statistical operations.
3Measurement precision
If multiple passes through the data are performed, then measurement precision is improved, but processing time increases
Solution Approach 1:
The patent performs preliminary computations of block means and sum of squares during a single pass through the data, storing these intermediate results for final variance calculation. This preliminary action eliminates the need for multiple passes while maintaining computational accuracy by preserving the necessary statistical moments.
Solution Approach 2:
The patent maintains continuous useful action by computing both the mean and sum of squares for each block in a single continuous operation during one data pass, rather than separating these computations into multiple passes. This continuity preserves accuracy while eliminating time loss from repeated data access.
Data Source
AI summary
A block-based iterative one-pass processing system and method for determining variance and mean is disclosed. Particularly, described is an approach for determining the variance of a data set that leverages a more efficient one-pass technique, that is tailored to maximize the processing capabilities of block-based hardware by processing an entire block of data at one time. Input data is separated into blocks of data and the block-based hardware processes each block until all of the blocks have been processed. For each block, computations may be performed to determine values specific to that block (e.g., mean, variance, etc.). Running values may be updated as each block is processed to include the resulting values from the current block.