Complex Event Processor Pulling Historical Data on Request
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Complex event processing systems face inefficiencies in combining and analyzing event data from various sources, as they often require replaying all historical data to meet query criteria, which is time-consuming and resource-intensive.
Innovation Solution
A complex event processor with a communications interface that retrieves and pushes event data only in response to request messages, utilizing an event processing pipeline with operator nodes and a combiner node to efficiently combine live and replayed event streams with historical data, leveraging reverse punctuation messages to retrieve relevant historical data just-in-time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all historical data is replayed to meet query criteria, then query completeness is improved, but processing time and resource consumption increase
Solution Approach 1:
The system pre-retrieves historical event data from storage before it is needed for query processing. By having the data ready in advance at the communications interface, the system eliminates the need to replay entire historical datasets when queries arrive, thus maintaining query completeness while reducing processing time.
Solution Approach 2:
The system extracts only the specific historical events that are relevant to the current query from the stored event data, rather than replaying all historical data. The communications interface identifies and retrieves only the necessary subset of events that match query criteria, reducing unnecessary processing while ensuring complete query results.
2Reliability
If all historical data is replayed to meet query criteria, then query completeness is improved, but resource consumption increases
Solution Approach 1:
The system extracts and retrieves only the specific historical events that are relevant to the current query from the stored event data, rather than replaying all historical data. The communications interface identifies and retrieves only the necessary subset of events that match query criteria, reducing unnecessary resource consumption while ensuring complete query results.
Solution Approach 2:
The system performs partial action by retrieving only the portion of historical data that is necessary to answer the query, rather than replaying the entire historical dataset. This partial retrieval approach reduces computational resources and energy consumption while still achieving complete and accurate query results.
3Reliability
If event data is retrieved and pushed continuously, then data availability is improved, but resource usage increases
Solution Approach 1:
The system maintains continuous readiness for data retrieval without continuous active retrieval. The communications interface remains ready to push events immediately when requested, and historical events are pre-loaded into memory, ensuring continuous data availability while avoiding the waste of continuously transferring data that would consume excessive resources.
Solution Approach 2:
The system uses pre-retrieved and cached event data at the communications interface to serve query requests immediately without needing to replay or re-retrieve data from storage each time. This self-service approach ensures data availability while minimizing resource usage by avoiding redundant data retrieval operations.
Data Source
AI summary
A complex event processor is described which has a communications interface configured to retrieve event data by pulling it from one or more sources and to receive at least one live event stream pushed to the interface. An event processing pipeline connected to the communications interface comprises a plurality of operator nodes connected between the communications interface and a combiner node which is a node configured to combine event data from the source and from the live event stream. The communications interface is configured to retrieve events from the source and to push the events retrieved from the source along the event processing pipeline towards the combiner node. The communications interface is configured to retrieve and push the retrieved events only in response to request messages passed in an upstream direction from the combiner node to the communications interface along the pipeline.