Multi-Agent Event Processing System for Real-Time Data Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing event processing systems face challenges in efficiently handling real-time and non-real-time data due to limitations in database access times, particularly for high data rate streams, making it difficult to analyze data in real-time.
Innovation Solution
A multi-agent based event processing system that includes a core engine for defining and deploying agents to perform programmable tasks with real-time data, a monitoring engine for lifecycle management, and a computing engine for execution, along with a batch processing system for non-real-time data, facilitated by a studio for user management and communication between systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored in a database before processing, then data can be retrieved and analyzed, but database access time limits real-time analysis capability
Solution Approach 1:
The system segments data processing into two distinct pathways: real-time stream processing and batch processing. Stream processing handles time-sensitive data directly without database storage, while batch processing handles non-time-sensitive data. This segmentation eliminates the bottleneck of database access for real-time analysis.
Solution Approach 2:
The patent introduces an event stream processor as an intermediary component that receives streaming data directly from data sources and processes it in real-time without requiring database storage. This intermediary bypasses the database access time constraint while maintaining data analysis capability.
2Productivity
If a single processing system is used, then system complexity is reduced, but the system cannot efficiently handle both real-time and non-real-time data requirements
Solution Approach 1:
The system dynamically routes data to appropriate processing pathways based on time sensitivity requirements. The architecture adapts its processing mode (stream or batch) according to the specific data characteristics and processing requirements, optimizing efficiency without requiring a completely different system for each case.
Solution Approach 2:
The patent creates a universal processing platform that can handle both real-time stream processing and batch processing through a unified architecture. The system includes components that can operate in multiple modes, eliminating the need for separate specialized systems while maintaining processing efficiency for different data types.
3Ease of operation
If database storage is used for all data, then data management is simplified, but high data rate streams cannot be analyzed in real-time
Solution Approach 1:
The system segments data management into stream processing (no database storage required) and batch processing (database storage used). This segmentation allows high data rate streams to be processed at line speed without database bottlenecks, while still providing simplified data management through the batch processing pathway for historical and non-time-sensitive data.
Data Source
AI summary
An event processing system includes a multi-agent based system, which includes a core engine configured to define and deploy a plurality of agents configured to perform a first set of programmable tasks defined by one or more users. The first set of tasks operates with real time data. The multi-agent based system also includes a monitoring engine configured to monitor a lifecycle of the agents, communication amongst the agents and processing time of the tasks. The multi-agent based system further includes a computing engine coupled to the core engine and configured to execute the first set of tasks. The event processing system includes a batch processing system configured to enable deployment of a second set of programmable tasks that operates with non-real time data and a studio coupled to the multi-agent based system and configured to enable users to manage the multi-agent based system and the batch processing system.