Flow Group Event Processing for Scalable Distributed Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed processing systems face challenges in providing efficient integration between applications while minimizing costs associated with provisioning, development, and maintenance, particularly in cloud computing environments where traditional methods are costly and inflexible.
Innovation Solution
The method involves grouping processing flows into flow groups with unique identifiers, using a messaging system to transmit event messages with associated identifiers, and deploying processing flows on-demand, allowing for stateless and scalable event processing without prior knowledge of message types, thereby reducing costs and enhancing flexibility.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional hardware processing is used for integration flows, then processing capability is provided, but costs (operational, development, maintenance) increase and flexibility decreases
Solution Approach 1:
The patent uses virtualization to create virtual copies of processing resources through flow groups and flow templates. Instead of provisioning physical hardware for each integration flow, the system creates virtual representations that can be instantiated and managed software-defined, reducing hardware costs while maintaining processing capability
Solution Approach 2:
The flow engine is designed to be universal and multi-functional, capable of executing any integration flow by loading appropriate flow templates. A single flow engine can handle multiple different integration scenarios by dynamically loading flow definitions, eliminating the need for dedicated hardware for each function
2Reliability
If processing resources are provisioned for all possible event types, then service quality is maintained, but resource provisioning costs increase
Solution Approach 1:
The system dynamically provisions processing resources based on actual event types and flow group identifiers in incoming messages. Rather than statically allocating resources for all possible events, the flow engine dynamically loads and executes only the necessary flow templates when specific event types occur, optimizing resource utilization
Solution Approach 2:
Flow templates are pre-defined and stored in the system, containing the processing logic for various event types. When an event occurs, the system quickly retrieves and executes the appropriate pre-prepared flow template, ensuring immediate response without needing to provision resources in advance for every possible scenario
3Adaptability or versatility
If integration flows are implemented with full customization, then specific processing requirements are met, but development and maintenance costs increase
Solution Approach 1:
Integration flows are segmented into reusable flow templates that can be independently defined, stored, and executed. Each flow template represents a modular unit of processing logic that can be combined and reused across different event types, reducing development effort compared to creating custom processing logic for each scenario
Data Source
AI summary
A facility for processing an event of a distributed processing environment is provided. The facility includes separating a plurality of processing flows into a plurality of flow groups, each flow group having a respective identifier. The identifier of each flow group is then associated with a respective topic. Based on the occurrence of an event, a message having a topic based on the event is published along with the identifier of the flow group associated with the topic. A processing flow of the flow group having the identifier of the message may then be used to process the event.