Data Estimation via Slope Correlation and Regression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The high storage costs and retention policies lead to the discarding of fine-grained historical data, which is essential for accurate forecasting and analysis, as it consumes more storage space than coarser-grained data, making it unavailable for systems that rely on historical data for forecasting and decision-making.
Innovation Solution
The technology estimates historical fine-grained data by calculating and correlating slopes from existing fine-grained data, applying these slopes to coarser-grained data within defined time periods, and adjusting for seasonality to regenerate the fine-grained data, ensuring data availability for forecasting models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If fine-grained data is stored at frequent intervals, then data accuracy and reliability improve, but storage costs increase significantly
Solution Approach 1:
The patent creates estimated copies of fine-grained data points using regression analysis on coarser data. Instead of storing all original fine-grained data, the system generates synthetic data points that replicate the essential patterns and trends, maintaining data accuracy while dramatically reducing storage requirements.
Solution Approach 2:
The patent transforms data from fine-grained to coarser-grained representation by changing the temporal sampling parameter. By storing data at lower resolution (coarser granularity) and using statistical methods to reconstruct fine-grained values when needed, the system reduces storage volume while preserving analytical accuracy.
2Quantity of substance
If retention policies discard fine-grained data to reduce storage costs, then storage costs decrease, but data availability for forecasting deteriorates
Solution Approach 1:
The patent performs preliminary regression analysis on available coarser-grained data to establish predictive models before fine-grained data is needed. These pre-computed models and patterns are stored, enabling rapid reconstruction of fine-grained data points when required for forecasting, thus preventing information loss despite data discarding.
Solution Approach 2:
The patent introduces regression analysis models as an intermediary between coarser stored data and the required fine-grained data. These models act as mediators that translate available coarse data into accurate fine-grained estimates, preserving data availability without requiring actual fine-grained data storage.
3Quantity of substance
If data is purged from the data store, then storage costs are reduced, but the ability to support forecasting models deteriorates
Solution Approach 1:
The patent creates synthetic copies of fine-grained data through regression analysis on coarser data. These estimated data points are generated on-demand to support forecasting models, maintaining forecasting reliability without requiring actual historical fine-grained data to be retained in the data store.
Solution Approach 2:
The patent replaces the mechanical storage and retrieval of actual fine-grained data with a computational system that generates estimates using regression analysis. This substitution transforms the system from physically storing all data to computationally reconstructing needed data, reducing storage requirements while maintaining forecasting capability.
Data Source
AI summary
A method is provided for estimating past data by identifying a high frequency data set for a defined time period. A pattern is calculated for the high frequency data set and then the pattern is applied to a low frequency data set in a past time period to estimate a high frequency query point.