Decremental Linear Regression Coefficient Calculation for Streamed Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for processing and analyzing Big Data, particularly streamed data, are inefficient and costly, as they require extensive computational resources and recalculate simple linear regression coefficients from scratch upon data changes, leading to time and resource wastage.
Innovation Solution
A computing system that maintains a computation set size counter and decrementally calculates simple linear regression coefficients by modifying existing components based on previous calculations, avoiding the need to revisit all data elements and store entire datasets, thus optimizing resource usage and reducing power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional methods recalculate simple linear regression coefficients from scratch when data changes, then calculation accuracy is maintained, but computational time and resource consumption increase significantly
Solution Approach 1:
The patent pre-calculates and stores summary statistics (sum of x, sum of y, sum of xy, sum of x², sum of y²) during the initial regression analysis. When data changes occur, these pre-computed values are adjusted incrementally rather than recalculating from all raw data points, significantly reducing computational time while maintaining accuracy
Solution Approach 2:
The patent divides the regression calculation into independent components (sum of x, sum of y, sum of xy, etc.) that can be updated separately. This segmentation allows the system to recalculate only the affected components when data changes, rather than recomputing the entire regression model from scratch
2Reliability
If all data elements are accessed and stored for regression calculation, then complete analysis is achieved, but memory requirements and storage capacity increase
Solution Approach 1:
The patent extracts only the essential summary statistics (aggregated values like sum of x, sum of y, sum of xy) needed for regression calculation, rather than storing and processing all individual data points. This extraction reduces memory requirements from O(n) to O(1) while preserving the ability to perform complete regression analysis
Solution Approach 2:
The patent creates and maintains copies of aggregated data (summary statistics) that can be updated incrementally. These copied aggregated values replace the need to store and access the original large dataset, reducing storage capacity requirements while maintaining analysis capability
3Productivity
If repeated recalculation is performed on large datasets, then updated results are obtained, but energy consumption and computational cost increase
Solution Approach 1:
The patent pre-computes and stores summary statistics that can be quickly adjusted when data changes. This preliminary action eliminates the need for energy-intensive full dataset processing during updates, reducing computational cost while maintaining fast result update speed
Solution Approach 2:
The patent changes the representation of data from individual data points to aggregated summary statistics. This parameter change allows for efficient incremental updates with minimal computational energy, as adjusting aggregated values requires far fewer operations than reprocessing entire datasets
Data Source
AI summary
The present invention extends to methods, systems, and computing system program products for decrementally calculating simple linear regression coefficients for Big Data or streamed data. Embodiments of the invention include decrementally calculating one or more components of simple linear regression coefficients for a modified computation set based on the one or more components of simple linear regression coefficients calculated for a previous computation set and then calculating the simple linear regression coefficients for the modified computation set based on the decrementally calculated components. Decrementally calculating simple linear regression coefficients avoids visiting all data elements in the modified computation set and performing redundant computations thereby increasing calculation efficiency, saving computing resources and reducing computing system's power consumption.