Stream Computing System Middleware-Free Architecture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Stream computing systems, such as those using the Storm system, face challenges in providing open services due to tight coupling with messaging middleware, leading to high recovery costs and potential avalanche effects during failovers, which limit horizontal scalability.
Innovation Solution
The system omits messaging middleware by supporting push and pull data access methods, using a data receiving module to write stream data into a file system and send blocks to a logic processing module for effective data determination and storage, ensuring data integrity and rapid recovery through checkpointing and RedoLog mechanisms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the Storm system uses source retransmission messaging mechanism during failover, then data can be recovered, but recovery cost increases and avalanche effect may occur
Solution Approach 1:
The patent implements checkpointing mechanisms that periodically save processing state and position information before failures occur. This preliminary action enables the system to resume from the last checkpoint rather than retransmitting all data from the source, significantly reducing recovery cost while maintaining data reliability.
Solution Approach 2:
The patent extracts and stores essential recovery information (checkpoint positions, processing state) separately from the main data stream. This extracted information is saved in the file system, allowing the system to recover without retransmitting the entire data stream from the source, thus reducing recovery cost while maintaining reliability.
2Reliability
If the Storm system uses source retransmission messaging mechanism, then data recovery is possible, but horizontal scalability is limited
Solution Approach 1:
The patent introduces a file system as an intermediary between the data source and processing nodes. This intermediary stores data blocks and checkpoint information, enabling multiple processing nodes to independently recover from failures without coordinating with the source or each other, thus improving horizontal scalability while maintaining data recovery capability.
Solution Approach 2:
The patent segments the data stream into discrete blocks that are independently stored in the file system. Each processing node can independently access and recover these blocks without requiring source retransmission, enabling scalable horizontal expansion while maintaining reliability through independent recovery paths.
3Ease of operation
If messaging middleware is used in the Storm system, then data transmission is enabled, but the system becomes tightly coupled and cannot provide open services
Solution Approach 1:
The patent extracts the data transmission and storage functionality from proprietary messaging middleware and implements it using standard file system operations. This extraction enables the system to provide open services through standard interfaces while maintaining data transmission capability, eliminating the tight coupling with specific middleware.
Solution Approach 2:
The patent uses a universal file system interface that can serve multiple purposes: data storage, checkpointing, and inter-node communication. This multi-functional approach replaces specialized messaging middleware, enabling the system to provide open services while maintaining all necessary data transmission capabilities through a universal interface.
Data Source
AI summary
System and method are disclosed for stream computing. An exemplary method may include receive data from a data processing module and determining whether the received data are effective data that are neither incomplete nor duplicative. The method may also include obtaining the effective data when it is determined that the received data are either incomplete or duplicative. In addition, the method may include storing the effective data in a log file of a file system.