Computational Node Intermediate Event Handling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Hadoop technology faces challenges in real-time performance due to the complexity of Map Reduce operations required for association computations, leading to bottlenecks in data processing and timeliness issues.
Innovation Solution
A method where a business processor entity registers predefined algorithms with a complex event processing instance, allowing for efficient computation and transfer of events based on triggering conditions, reducing the need for intermediate file storage and improving timeliness by directly processing and transferring computation results within the system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If Map Reduce operations are performed multiple times for association computation in Hadoop, then computation completeness is improved, but processing time and IO operations increase significantly
Solution Approach 1:
The patent extracts the intermediate result writing operation from the Map Reduce workflow. Instead of writing intermediate results to HDFS and reading them back for subsequent computations, the system keeps intermediate results in memory within the same computing node, eliminating unnecessary IO operations and reducing processing time while maintaining computation completeness
Solution Approach 2:
The patent introduces a computing node as an intermediary that performs multiple Map Reduce operations in sequence without writing intermediate results to external storage. The computing node acts as a mediator that holds intermediate results in memory, allowing subsequent MR operations to access them directly, thus avoiding the HDFS write-read cycle
2Reliability
If intermediate results are written to HDFS for each Map Reduce operation, then data persistence is improved, but IO operations become a bottleneck
Solution Approach 1:
The patent applies local quality by making the storage characteristic location-dependent. Intermediate results are stored in volatile memory when located within the same computing node, providing fast access. Only final results are persisted to HDFS, optimizing the balance between persistence and access speed based on the local context of data location
Solution Approach 2:
The patent segments the data lifecycle into two distinct phases: intermediate results remain in memory during the computation phase for fast access, while only final results are persisted to HDFS in the output phase. This segmentation allows the system to optimize for speed during processing and for persistence only when necessary
3Measurement precision
If multiple Map Reduce operations are performed for event association, then computation accuracy is improved, but real-time performance deteriorates
Solution Approach 1:
The patent applies preliminary action by pre-loading event data into memory before performing Map Reduce operations. This preliminary preparation allows subsequent computations to access data rapidly from memory rather than from disk, maintaining computation accuracy while significantly improving real-time performance
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Embodiments of the present invention provide a method for processing data, a computing node, and a system. The method includes: registering, by a BPE, an algorithm with a CEP instance; transferring, by the CEP instance when detecting that an event concerned by the algorithm satisfies a computation-triggering condition, an event required for computation to the BPE; obtaining, by the BPE, a computation result, and if determining that a further computation is required for the computation result, writing the computation result as an intermediate event to the CEP instance; and transferring, by the CEP instance when detecting that an event concerned by another algorithm satisfies a computation-triggering condition thereof and the intermediate event is an event required for computation thereof, the intermediate event to a BPE that registers the another algorithm. The CEP instance performs association of multiple events and multiple algorithms, which simplifies a computation process and improves timeliness.