Stream Processing Missing Data Correction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Stream processing systems face challenges in handling missing data parts during real-time processing, which can lead to inaccurate results and require extensive reprocessing time, especially in applications like vehicle test driving where timely analysis is critical.
Innovation Solution
A method and system for detecting missing parts in stream data, comparing them with master data, and updating the processing results using a second stream processing line, allowing for real-time correction and reprocessing with limited system resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If stream processing is performed in real time on received data, then processing speed is improved, but accuracy deteriorates due to missing data parts
Solution Approach 1:
The system performs preliminary detection of missing data parts during stream processing, identifies which data elements are absent before final processing completes, and prepares for correction by comparing with master data. This preliminary identification allows the system to maintain real-time processing speed while planning accuracy corrections.
Solution Approach 2:
The system implements a feedback mechanism where processing results are continuously compared between the stream processing line (using received data) and the master data processing line. When discrepancies are detected due to missing data, the system feeds back correction information to update the stream processing results, thereby improving accuracy while maintaining real-time performance.
2Measurement precision
If reprocessing is performed to correct missing data, then accuracy is improved, but processing time increases
Solution Approach 1:
Instead of completely reprocessing all data when missing parts are detected, the system performs partial reprocessing only on the specific portions affected by missing data. It identifies the exact operators and data segments that need correction and applies updates only there, significantly reducing the time penalty while maintaining accuracy improvements.
Solution Approach 2:
The system detects missing data parts during the ongoing stream processing operation, before final results are committed. By identifying issues preliminarily and using master data for correction, the system avoids the need for complete reprocessing after the fact, thus reducing time loss while improving accuracy.
3Measurement precision
If complete data is processed, then accuracy is improved, but system resource consumption increases
Solution Approach 1:
The system uses master data selectively and partially - only for the specific data parts identified as missing or problematic in the received stream data. Rather than processing complete master data sets repeatedly, it applies corrections only where needed, reducing system resource consumption while maintaining data integrity where it matters most.
Solution Approach 2:
The system applies different processing qualities to different parts of the data stream. For complete received data, standard stream processing is used. For parts identified as missing or problematic, enhanced processing using master data is applied locally. This local quality approach ensures data integrity where needed while conserving system resources overall.
Data Source
AI summary
A method for stream-processing data including a missing part in real time and thereafter updating the result of the stream processing. A technique for processing data is included. The technique includes receiving data; detecting a probably missing part in the received data while stream-processing the received data in real time; and comparing master data corresponding to the received data and having no missing part with the probably missing part, and if the received data has the missing part, updating the result of the stream processing using the master data.