Multi-Channel Stream Reassembly With History-Based Data Scrubbing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data processing systems face limitations in handling high sustained data rates due to differences in bandwidth between input/output interfaces and processors, leading to inefficiencies in processing large amounts of data, such as in real-time facial recognition applications, where accurate identification of individuals is hindered by the system's inability to handle constant data flow effectively.
Innovation Solution
A method and system for processing streaming data at high sustained rates by receiving data elements over non-sequential communication channels, writing them directly to a file system in an unassembled manner, and performing data scrubbing operations to identify and reassemble missing data elements, utilizing history information to determine when scrubbing is necessary, thereby optimizing data transfer and handling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is transmitted over multiple non-sequential communication channels to increase bandwidth, then sustained data rate is improved, but data reassembly complexity and processing overhead increase
Solution Approach 1:
The patent divides the data stream into multiple segments transmitted over separate communication channels. Each channel carries portions of the overall data flow, allowing parallel transmission and increasing total bandwidth utilization. The data is segmented at the source and reassembled at the destination, enabling high sustained data rates while maintaining manageable complexity through structured segmentation.
Solution Approach 2:
The patent introduces a temporal dimension to data transmission by using non-sequential channels with varying latency characteristics. Data elements are transmitted across multiple dimensions of communication resources (different channels, different time slots) and reassembled based on their logical sequence rather than transmission order, effectively adding dimensional flexibility to the transmission system.
2Reliability
If data scrubbing operations are performed to ensure data completeness, then data reliability is improved, but processing time and computational overhead increase
Solution Approach 1:
The patent performs preliminary data scrubbing operations during the data transmission and reception process itself, rather than as a separate post-processing step. By integrating scrubbing operations into the data flow pipeline, the system validates data completeness concurrently with reception, reducing overall processing time while maintaining reliability.
Solution Approach 2:
The patent implements feedback mechanisms where the system monitors data reception status and dynamically adjusts scrubbing operations based on observed data flow patterns. History information about sequential vs. non-sequential data arrival is used to intelligently determine when scrubbing is necessary, reducing unnecessary processing while ensuring data completeness when needed.
3Productivity
If data is written directly to file system without pre-processing to maintain high data rates, then productivity is improved, but data integrity and completeness become harder to ensure
Solution Approach 1:
The patent introduces an intermediary buffering layer between the communication channels and the file system. This intermediary structure receives data elements from multiple channels, performs preliminary organization and validation, then writes to the file system in an optimized manner. The intermediary maintains data integrity through structured storage while preserving high write rates by avoiding complex real-time reassembly before file writing.
Data Source
AI summary
Mechanisms are provided for processing streaming data at high sustained data rates. These mechanisms receive a plurality of data elements over a plurality of non-sequential communication channels and write the plurality of data elements directly to the file system of the data processing system in an unassembled manner. The mechanisms determining whether to perform a data scrubbing operation or not based on history information indicative of whether data elements in the plurality of data elements are being received in a substantially sequential manner. The mechanisms perform a data scrubbing operation, in response to a determination to perform data scrubbing, to identify any missing data elements in the plurality of data elements written to the file system and assemble the plurality of data elements into a plurality of data streams in response to results of the data scrubbing indicating that there are no missing data elements.