Semantic Layer Continuous Query System for Low Latency Event Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing business intelligence and enterprise systems face challenges in efficiently processing large volumes of data and providing real-time notifications to a large number of users with low latency, especially when dealing with complex event streams and queries, which can be difficult for non-technical users to establish and manage.
Innovation Solution
A semantic layer-based system that uses a server-side event processor to create and manage continuous queries, allowing users to define queries through a graphical interface, which automatically generates Event Processing Language (EPL) statements to monitor and filter event streams, and push relevant information to clients using a push-model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a client side pull-based architecture is used to retrieve business information, then users can access data from servers, but it becomes difficult to scale for hundreds of users when tens of thousands of events occur per second and cannot provide low latency notifications
Solution Approach 1:
The patent inverts the traditional pull-based architecture by implementing a push-based architecture where the server proactively pushes business information to clients based on subscribed criteria. This inversion eliminates the need for continuous client polling, dramatically improving scalability to handle hundreds of users with tens of thousands of events per second while maintaining low latency notifications.
Solution Approach 2:
The patent introduces a semantic layer as an intermediary component that sits between the event stream and the push notification system. This semantic layer processes events, applies user-defined criteria, and determines which information should be pushed to which clients. This intermediary enables efficient filtering and routing, making the system scalable while maintaining productivity.
2Loss of information
If complex queries are established to monitor event streams, then specific business information can be retrieved, but the process becomes time consuming and error prone for business users without programming experience
Solution Approach 1:
The semantic layer acts as an intermediary that translates user-friendly subscription criteria into the complex Event Processing Language (EPL) statements required by the underlying system. Business users interact with the system through simple, intuitive criteria definitions rather than writing complex queries directly, eliminating the time-consuming and error-prone process while maintaining accurate information retrieval.
Solution Approach 2:
The patent segments the query creation process into distinct layers: users define high-level subscription criteria in the semantic layer, which then automatically generates and manages the complex EPL statements. This segmentation allows business users to focus on business logic while the system handles technical complexity, improving ease of operation without sacrificing information accuracy.
3Loss of information
If large amounts of data are processed to provide notifications to a large number of users, then comprehensive business information is available, but latency increases and cannot be maintained at a few milliseconds
Solution Approach 1:
The semantic layer intermediary efficiently processes large volumes of events by applying pre-defined subscription criteria and maintaining indexes of relevant data. This intermediary architecture enables the system to process comprehensive business information while maintaining low latency by avoiding full data scans and enabling direct matching of events against user subscriptions.
Solution Approach 2:
The system performs preliminary actions by pre-processing and indexing business information in the semantic layer according to user subscriptions before events arrive. This preliminary organization of data enables rapid matching and notification delivery, maintaining latency at a few milliseconds even when processing large amounts of comprehensive business information for numerous users.
Data Source
AI summary
A system, method, means, and/or computer program code may be provided to facilitate a presentation of information to a user. According to some embodiments, continuous query definition parameters are created by a user via a graphical user interface. Moreover, semantic layer information associated with an event processing engine may be provided. The event processing engine may, for example, receive an event stream from a data source. According to some embodiments, the semantic layer information is defined by a semantic layer designer via a graphical user interface. At least one event processing language statement may be created based on (i) the continuous query definition parameters and (ii) the semantic layer information. The statement may then be provided to the event processing engine so as to establish a continuous query at the event processing engine, the continuous query providing output data based on events in the event stream. According to some embodiments, a continuous meta-query for a complex event processing engine is initially created. When a connection with a business user is subsequently established, a continuous user-query is initiated based on the continuous meta-query.