Stream Message Linearization With Staged Data Normalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing stream-processing systems struggle to efficiently handle and process data objects from diverse external data sources, which can be structured in various formats and protocols, leading to challenges in data handling and processing efficiency.
Innovation Solution
A distributed stream-processing system that linearizes the processing of data objects by categorizing tasks into sequences, including in-protocol, unbundling, unpacking, message normalization, and validation categories, allowing for efficient handling and processing of data objects from various sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the system processes data objects directly from diverse external data sources, then processing speed is maintained, but the system complexity increases due to needing to handle multiple protocols and formats
Solution Approach 1:
The patent introduces a message normalization task as an intermediary component between diverse data sources and downstream processing tasks. This normalization layer receives data in various formats (JSON, XML, CSV, etc.) and converts them into a unified internal representation, thereby reducing system complexity while maintaining processing speed through centralized format handling.
Solution Approach 2:
The processing pipeline is segmented into distinct functional categories: in-protocol tasks for receiving data, unbundling tasks for extracting files, unpacking tasks for decompressing, message normalization tasks for format conversion, and validation tasks for data verification. This segmentation allows each component to specialize in handling specific aspects of data processing, reducing overall system complexity.
2Adaptability or versatility
If the system accepts messages from wide range of protocols and formats, then adaptability improves, but processing efficiency decreases due to the need to handle varying data structures
Solution Approach 1:
The message normalization task serves as a universal interface that handles multiple data formats (JSON, XML, CSV, HTML, etc.) and protocols through a single standardized processing mechanism. This multi-functional approach maintains adaptability to diverse data sources while improving processing efficiency by consolidating format conversion logic in one location rather than requiring separate handling for each format.
Solution Approach 2:
The system changes the parameter of data representation by converting all incoming data from various formats into a standardized internal representation format. This parameter transformation allows the downstream processing tasks to operate efficiently on uniform data structures without needing to handle the complexity of multiple external formats directly.
3Loss of time
If the system processes data objects immediately, then response time is reduced, but reliability decreases when tasks cannot operate on certain data objects
Solution Approach 1:
The validation task category provides feedback mechanisms that verify data object properties before passing them to downstream tasks. This feedback loop ensures that only data objects meeting specific criteria are processed, preventing failures downstream while maintaining fast processing by validating data upfront rather than attempting processing and handling failures later.
Solution Approach 2:
The system performs preliminary validation and normalization actions before data objects reach processing tasks. By preparing and verifying data in advance through the message normalization and validation tasks, the system ensures task reliability is maintained while response time is minimized, as the data is already prepared and validated when it reaches the processing stage.
Data Source
AI summary
A data object from a data source is received by a distributed process in a data stream. The distributed process has a sequence of categories, each category containing one or more tasks that operate on the data object. The data object includes files that can be processed by the tasks. If the task is able to operate on the data object, then the data object is passed to the task. If the task is unable to operate on the data object, then the files in the data object are passed to a file staging area of the distributed process and stored in memory. The files in the file staging area are passed, in sequence, from the file staging area to the task that was unable to operate on the data object. The data object is outputted to a next category or data sink after being operated on by the task.