Iterative Simple Linear Regression Coefficient Calculation for Big Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for processing Big Data, particularly in real-time decision-making scenarios like the financial industry, are inefficient and costly due to the need to recalculate simple linear regression coefficients for large datasets after data changes, leading to slow response times and wasteful use of computing resources.
Innovation Solution
A method for iteratively calculating simple linear regression coefficients by modifying existing components based on previous calculations, avoiding the need to access all data elements in the computation set and reducing redundant computations, thereby enhancing efficiency and resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional simple linear regression coefficient calculation is performed on all data elements after data changes, then calculation accuracy is maintained, but computational load and time consumption increase significantly
Solution Approach 1:
The patent divides the computation set into a buffer containing a subset of data elements and excludes other data elements. Instead of recalculating regression coefficients using all data elements, the method performs calculations only on the buffered subset, significantly reducing computational load while maintaining acceptable accuracy for real-time decision-making.
Solution Approach 2:
The patent applies partial action by calculating regression coefficients using only a portion of the available data elements (those in the buffer) rather than all data elements. This partial calculation approach provides sufficient accuracy for real-time applications while dramatically improving calculation efficiency and reducing resource consumption.
2Quantity of substance
If all data elements are accessed for regression coefficient calculation, then complete data utilization is achieved, but resource waste and power consumption increase
Solution Approach 1:
The computation set is segmented into a buffer containing selected data elements and excluded data elements. The method accesses and processes only the buffered subset, reducing the quantity of data elements that need to be read from storage and processed, thereby lowering power consumption and resource utilization while maintaining acceptable calculation quality.
3Measurement precision
If computation set size is increased to include more data elements, then calculation accuracy improves, but response time and processing speed decrease
Solution Approach 1:
The patent uses partial action by selecting a buffer size that provides sufficient calculation accuracy for real-time decision-making without requiring access to all available data elements. This optimized buffer size achieves the right balance between accuracy and response time, enabling faster processing speeds while maintaining acceptable regression coefficient accuracy.
4Reliability
If traditional recalculation method is used after data changes, then up-to-date coefficients are obtained, but computing resource waste increases
Solution Approach 1:
The patent segments the computation set into a manageable buffer and excluded data elements. When data changes occur, the method updates regression coefficients by recalculating only using the buffered subset rather than all data elements. This approach maintains data freshness and reliability of coefficients while significantly reducing computing resource waste and power consumption.
Data Source
AI summary
Methods, systems, and computing system program products for iteratively calculating Simple Linear Regression (SLR) coefficients for Big Data, including iteratively calculating one or more components of SLR coefficients for a modified computation set based on one or more components of SLR coefficients calculated for a pre-modified computation set and then calculating the SLR coefficients for the modified computation set based on the iteratively calculated components. Iteratively calculating SLR coefficients avoids visiting all data elements in the modified computation set and performing redundant computations thereby increasing calculation efficiency, saving computing resources and reducing computing system's power consumption.