Event Data Deserialization Filtering for Distributed Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed-parallel complex event processing systems, the existing techniques lead to increased load on nodes due to unnecessary deserialization of event data, even when data is to be discarded, as all event data is deserialized during preprocessing, without determining whether it should be discarded.
Innovation Solution
A processing system that reconstitutes only a specific portion of received data based on reference information for query processing, determining whether to discard the data, and performing full deserialization only when necessary, thereby reducing unnecessary deserialization and load on nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all event data is deserialized during preprocessing, then the system can determine whether to process or discard the data, but the load on nodes increases due to unnecessary deserialization of data that will be discarded
Solution Approach 1:
The patent segments the deserialization process into two distinct phases: (1) deserializing only the specific portion of event data needed for preprocessing filtering, and (2) deserializing the remaining data only if it passes the filter. This segmentation allows the system to avoid deserializing entire event objects when only specific fields are needed for filtering decisions, thereby reducing unnecessary processing load while maintaining accurate filtering capability.
Solution Approach 2:
The patent applies partial action by deserializing only the necessary portion of event data during preprocessing rather than the entire event object. The system extracts and deserializes only the specific fields required for filtering conditions, leaving the rest of the event data in serialized form until it is confirmed to need full processing. This partial deserialization approach significantly reduces the processing load on nodes while maintaining the ability to accurately determine whether to process or discard event data.
2Productivity
If event data is not deserialized, then the load on nodes is reduced, but the system cannot determine whether to discard the event data in preprocessing
Solution Approach 1:
The patent segments event data into two parts: (1) specific portion containing fields needed for preprocessing filtering, and (2) remaining portion containing other fields. The system deserializes only the specific portion during preprocessing to enable filtering decisions, while keeping the remaining portion serialized. This segmentation ensures that filtering accuracy is maintained through proper deserialization of critical fields, while overall processing efficiency is improved by avoiding deserialization of the entire event object.
Solution Approach 2:
The patent performs preliminary deserialization of only the necessary fields before the main processing decision is made. By extracting and deserializing specific portions of event data in advance, the system enables accurate filtering decisions to be made on serialized data, and only then proceeds to deserialize the full event object if needed. This preliminary partial deserialization action maintains filtering accuracy while minimizing the scope of expensive deserialization operations.
3Adaptability or versatility
If full deserialization is performed on all event data, then complete data is available for processing, but unnecessary deserialization increases the load on nodes
Solution Approach 1:
The patent implements a dynamic deserialization strategy where the extent of deserialization is adjusted based on the specific needs of each event. The system first performs minimal deserialization of specific fields to determine processing requirements, then dynamically decides whether full deserialization is necessary. This dynamic approach allows the system to maintain data processing flexibility by deserializing only what is needed for each individual event, rather than applying a static full-deserialization approach to all events, thereby improving node processing efficiency.
Solution Approach 2:
The patent changes the parameter of deserialization completeness from a fixed state (always fully deserialized) to a variable state (partially or fully deserialized based on needs). By controlling the deserialization parameter dynamically - deserializing only specific portions when sufficient for filtering, and fully deserializing only when necessary - the system maintains adaptability in data processing while significantly reducing the processing load on nodes through parameter optimization.
Data Source
AI summary
A non-transitory, computer-readable recording medium having stored therein a processing program causes a computer to execute a process including reconstituting a specific portion of received data, based on reference information specifying a referenced portion of the received data for selecting processing data from the received data, the processing data being subject to a query processing; determining whether to discard the received data, based on the specific portion of the received data and a selecting condition for selecting the processing data; and reconstituting the received data when the determining determines not to discard the received data.