Embedded Rule Execution Operator for Streaming Data Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Streaming platforms face challenges in efficiently processing and transmitting large volumes of raw data, leading to computational resource exhaustion, memory limitations, and latency due to the need to transmit enriched data externally for further processing, which introduces data loss and delays.
Innovation Solution
Integration of a custom-built, embedded rule execution operator within the streaming platform to process and implement actions in real-time, reducing the need for external data transmission and enhancing resource efficiency by directly executing rules on enriched data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If streamed data is transmitted to external systems for processing, then data can be further analyzed and processed, but computational resources are exhausted and latency increases
Solution Approach 1:
The patent merges the rule execution operator directly into the streaming platform, combining data enrichment and rule execution functions into a single integrated system. This eliminates the need to transmit enriched data externally for further processing, thereby reducing latency while maintaining full processing capability. The integrated operator processes rules against enriched information within the same platform that generates the enriched data.
2Productivity
If streamed data is transmitted externally, then further processing can occur, but data loss opportunities arise and bandwidth resources are consumed
Solution Approach 1:
By integrating the rule execution operator within the streaming platform, the patent eliminates data transmission to external systems. The enriched information remains within the platform throughout the entire processing pipeline, from enrichment through rule execution to event detection. This eliminates opportunities for data loss during transmission and removes bandwidth consumption associated with external data transfer.
3Measurement precision
If custom rule execution is implemented, then event detection accuracy improves, but system complexity increases
Solution Approach 1:
The patent integrates the rule execution operator directly into the streaming platform, merging custom rule execution capabilities with the existing data enrichment functionality. This integration allows the system to execute custom rules against enriched information with high accuracy while avoiding the added complexity of separate external processing systems. The unified architecture shares common infrastructure, data structures, and processing pipelines.
4Loss of information
If data is enriched and transmitted externally, then contextual information is added, but computational resources are depleted
Solution Approach 1:
The patent combines the rule execution operator with the streaming platform's data enrichment capabilities, creating a unified processing system. This integration allows contextual information to be added through enrichment and immediately utilized by the rule execution operator without requiring external transmission. The system maintains full contextual information while optimizing computational resource usage by processing everything within a single integrated platform.
Data Source
AI summary
The present invention is directed to a system and methods for integrating flexible event detection into a near real-time streaming environment. A streaming platform streams raw data from multiple sources, adds contextual information to the raw data, and makes inferences from the enriched information. A rule execution operator, being integrated within the streaming platform, executes rules against the enriched information to detect events, using a rule repository that stores a plurality of flexible, dynamic, and customizable rules. When an event is detected by the rule execution operator, the streaming platform may use a middleware component to instantiate actions that are responsive to a detected event. Actions may be directly performed, and/or the actions and instructions for implementing or performing those actions are communicated to external devices.