Time-Series Data Cleaning via Weekly Copying and Interpolation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In machine learning systems, particularly in hyper-converged infrastructure and time-series data, missing data points can lead to biases and reduced model effectiveness due to various factors like network issues or data collection problems, and determining the best method to handle these gaps is crucial for accurate predictions.
Innovation Solution
An information handling system configured to receive time-series data, identify missing points, and fill them by copying data from corresponding earlier or later weeks if available, or performing interpolation when neither is available, to maintain data representativeness and avoid biases.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If missing data points are removed from time-series data, then the data quality is improved, but the data completeness and representativeness deteriorate
Solution Approach 1:
The patent applies copying by duplicating data points from corresponding time periods in previous weeks to fill missing data gaps. When a data point is missing at a specific time, the system copies the corresponding data point from the same time in earlier weeks, preserving data completeness while maintaining temporal patterns and seasonality in the time-series data.
2Loss of information
If data copying from earlier weeks is used to fill missing points, then data completeness is improved, but temporal accuracy may deteriorate
Solution Approach 1:
The patent applies preliminary action by proactively filling missing data points before they are used in analysis or modeling. The system identifies missing points and replaces them with copied data from previous weeks in advance, ensuring that the dataset is complete and ready for subsequent processing without introducing temporal inaccuracies during critical analysis phases.
Solution Approach 2:
The patent applies parameter changes by dynamically adjusting the data filling strategy based on the specific characteristics of missing data. The system evaluates whether to copy data from earlier weeks or perform interpolation based on the pattern and extent of missingness, changing the approach parameters to optimize both completeness and temporal accuracy for different scenarios.
3Device complexity
If simple data copying is used to fill missing points, then the processing complexity is reduced, but the handling of complex missing patterns deteriorates
Solution Approach 1:
The patent applies dynamics by implementing a flexible, adaptive data filling approach that adjusts its complexity based on the specific missing data patterns encountered. The system dynamically switches between simple copying from previous weeks and more sophisticated interpolation methods depending on the nature and extent of missingness, optimizing the balance between processing complexity and reliability for different scenarios.
Data Source
AI summary
An information handling system may be configured to: receive time-series data regarding measurements of a physical variable; determine that a missing data point is missing from the time-series data, wherein the missing data point corresponds to a particular day of the week; in response to a determination that earlier corresponding data from the particular day of the week of an earlier week is available, copy the earlier corresponding data to replace the missing data; in response to a determination that later corresponding data from the particular day of the week of a later week is available, copy the later corresponding data to replace the missing data; and in response to a determination that neither the earlier corresponding data nor the later corresponding data is available, perform interpolation to replace the missing data.