Message Parsing in Distributed Stream Processing Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed stream processing systems, parsing messages for Internet of Things devices and big data analysis is inefficient due to high workload and increased memory usage and network bandwidth from processing unnecessary data fields, which hampers throughput and efficiency.
Innovation Solution
A method that identifies specific data fields useful for parsing, generates a mask sequence with relevant bits set to a specific value, and filters out less useful fields, reducing memory usage and network bandwidth by only processing and transmitting essential data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If all data fields are processed for message parsing, then parsing completeness is maintained, but processing workload and memory usage increase significantly
Solution Approach 1:
The patent extracts only the necessary data fields required for parsing from the complete message structure. By identifying and separating essential fields from unnecessary ones, the system processes only relevant data, thereby reducing memory consumption and processing workload while maintaining parsing completeness for the required fields.
Solution Approach 2:
The message data fields are segmented into essential fields and non-essential fields. The system then processes only the essential fields separately, dividing the parsing task into meaningful segments. This segmentation enables selective processing that reduces overall workload and memory usage while preserving the integrity of critical data.
2Productivity
If all data fields are transmitted in distributed processing, then data completeness is maintained, but network bandwidth consumption increases
Solution Approach 1:
The patent extracts and transmits only the essential data fields across the distributed system network. By removing non-essential fields from the transmission data, the system reduces network bandwidth consumption significantly while maintaining the completeness and integrity of the required data for processing decisions.
Solution Approach 2:
Instead of transmitting all data fields (excessive action), the system transmits only the partial set of essential fields needed for processing. This partial action approach optimizes network bandwidth usage by avoiding the transmission of redundant or unnecessary data while still providing sufficient information for effective data processing.
3Reliability
If unnecessary data fields are processed, then parsing completeness is maintained, but processing time and workload increase
Solution Approach 1:
The patent extracts and processes only the essential data fields that are necessary for accurate parsing. By removing unnecessary fields from the processing pipeline, the system maintains parsing accuracy for critical data while significantly reducing processing time and computational workload associated with handling redundant information.
Data Source
AI summary
Techniques for parsing a message comprising at least one data field are provided. One computer-implemented method comprises: determining, by an electronic device operatively coupled to a processing unit, a first model from a first category of data processing models; and determining, by the electronic device, a second model from a second category of data processing models, the second category being different from the first category. The computer-implemented method also comprises comparing, by the electronic device, performance of a first combined model of first and second models with performance of the first model; and selecting, by the electronic device, a target model for processing data from the first combined model and the first model based on the comparing.