Incremental Autoregression for Sensor Data Memory Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current autoregression processing systems require large memory resources, leading to high memory consumption and slow performance when processing large chunks of data, especially in IoT environments where multiple tasks are executed on the same machine.
Innovation Solution
The implementation of incremental autoregression techniques that process time series data by updating convolution values, partial sums, and observation counts, allowing for the generation of autoregression coefficients with reduced memory usage through efficient matrix computations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If entire time series data is loaded into memory for autoregression analysis, then analysis completeness is improved, but memory consumption increases significantly
Solution Approach 1:
The patent divides the time series data into blocks and processes them incrementally. Instead of loading the entire dataset into memory, the system processes data in manageable segments, updating convolution values and partial sums block by block. This segmentation allows complete analysis without requiring excessive memory resources.
Solution Approach 2:
The patent pre-computes and stores convolution values and partial sums in a compressed manner before final autoregression analysis. By performing preliminary calculations and storing only essential intermediate results rather than raw data, the system reduces memory requirements while maintaining analysis completeness.
2Measurement precision
If worker nodes load entire partition data into memory, then processing accuracy is improved, but system overhead increases and performance slows down
Solution Approach 1:
The system segments large datasets into smaller blocks that can be processed efficiently in memory. Each worker node processes blocks incrementally, maintaining processing accuracy through proper block management while avoiding the performance penalty of loading entire partitions into memory.
Solution Approach 2:
The patent changes the processing parameters from batch loading of entire datasets to incremental block processing. By adjusting the data processing approach to use block size parameters and incremental updates, the system maintains accuracy while significantly improving performance and reducing system overhead.
3Adaptability or versatility
If multiple autoregression jobs are executed on the same machine, then resource utilization is improved, but memory capacity is exceeded
Solution Approach 1:
The patent enables multiple autoregression jobs to run simultaneously by segmenting data processing into blocks. Each job processes its data blocks independently with minimal memory footprint, allowing high resource utilization without exceeding the machine's total memory capacity.
Solution Approach 2:
The system performs partial processing of data blocks in each job cycle, completing full analysis through multiple passes. This partial action approach allows multiple jobs to share memory resources efficiently, with each job using only the memory needed for its current block rather than requiring full dataset memory allocation.
Data Source
AI summary
This disclosure relates to systems and methods for analyzing sensor data using incremental autoregression techniques for generating a vector of autoregression coefficients is provided. The system processes a time series data to obtain blocks of observation values, reads the observation values, updates pre-stored convolution values with the observation values, updates a partial sum by adding each observation value to the partial sum, increments a count each time an observation value is read, repeats the steps of updates and increments until a last observation value from a last block is read to obtain an updated set of convolution values, partial sum, and count. The system further computes a first matrix and a second matrix using the updated set of convolutions values, or summation of observation values computed from the updated partial sum, or the updated count, and generates a vector of autoregression coefficients based on the first and the second matrix.