Data Stream Gap Filling Using Smallest-Sequence Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for handling missing data in data streams often produce inaccurate values for larger sequences due to poorer accuracy in missing data models and inefficient iterative estimation processes.
Innovation Solution
A method that processes the smallest sequences of missing data points first, using a missing data model to determine values, which are then used to correct subsequent sequences, with weighting based on distance and verification criteria to ensure accuracy and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing methods process all missing data points at once using iterative estimation, then the process covers the entire data stream, but the accuracy deteriorates for larger sequences of missing data
Solution Approach 1:
The patent segments the data stream into smaller windows or batches, processing missing data points in incremental portions rather than attempting to estimate all missing values simultaneously. This segmentation allows the model to maintain higher accuracy on smaller subsets while progressively filling the entire data stream.
Solution Approach 2:
The patent performs preliminary actions by first identifying and processing smaller, more certain missing data sequences before tackling larger ones. By establishing accurate baseline corrections early in the process, the system creates a foundation that improves subsequent estimations of larger missing data blocks.
2Reliability
If existing methods iteratively improve estimates for all missing data points, then comprehensive correction is achieved, but processing efficiency deteriorates
Solution Approach 1:
The patent divides the iterative estimation process into segmented passes over the data stream. Each pass processes a portion of missing data points using the missing data model, updating the corrected data stream incrementally. This segmentation reduces computational complexity and memory requirements compared to maintaining and iteratively updating estimates for all missing points simultaneously.
Solution Approach 2:
The patent performs preliminary processing by identifying and correcting smaller, more certain missing data sequences first. These preliminary corrections are then used as inputs for subsequent processing of larger or more uncertain missing data sequences, creating a cascading effect that improves overall efficiency and accuracy.
3Quantity of substance
If missing data models process larger sequences of missing data, then more complete coverage is achieved, but model accuracy deteriorates
Solution Approach 1:
The patent applies segmentation by processing missing data in smaller windows or batches rather than attempting to model entire large sequences at once. Each smaller segment can be accurately modeled using the missing data model, and the results are combined to achieve complete coverage while maintaining high accuracy throughout.
Solution Approach 2:
The patent performs preliminary corrections on smaller, more certain missing data sequences first. These preliminary results are then used as known values or constraints when processing larger sequences, effectively breaking down the difficult problem of modeling large missing sequences into manageable steps that maintain model accuracy.
Data Source
AI summary
There is described a method for communicating data, the method comprising: receiving an incomplete data stream, wherein the incomplete data stream comprises a plurality of sequences of data points having respective values and a plurality of sequences of missing data points; receiving a missing data model; determining values for each of the plurality of sequences of missing data points, comprising: selecting a sequence of missing data points that has not previously been processed, wherein the sequence of missing data points to be processed is selected as a smallest sequence of missing data points of the plurality of sequences of missing data points that have not previously been processed; processing the incomplete data stream to determine values for the selected sequence of missing data points based upon the missing data model; updating the incomplete data stream to include the determined values for the selected sequence of missing data points; and wherein values for subsequent sequences of missing data points are generated based upon the updated data stream; and outputting a corrected data stream comprising the determined values for each of the plurality of sequences of missing data points.