Distributed Event Streaming System for Low-Latency Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large networked environments face challenges in detecting and responding to events due to latency and overwhelming computing resources, particularly in processing and storing the associated data.
Innovation Solution
A distributed event streaming system that includes controllers for real-time event detection and data processing, with sensitive information removal, and a central system featuring a repository cluster with both relational and NoSQL databases for efficient event handling and storage, utilizing load balancers and enterprise service buses for scalable event routing and processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large volumes of event data are transmitted and stored in a centralized system, then complete event detection and storage capability is achieved, but computing resources are overwhelmed and latency increases
Solution Approach 1:
The system divides the centralized event processing architecture into distributed edge computing nodes deployed across multiple locations. Each node independently processes and filters events locally, segmenting the computing workload and preventing any single system from being overwhelmed by the entire volume of event data.
Solution Approach 2:
The patent introduces a hierarchical dimension to the architecture, transitioning from a single centralized layer to multiple layers including edge nodes, regional aggregation points, and central repositories. This dimensional expansion distributes data flow and processing across different levels, reducing the burden on any single component.
2Loss of information
If all event data is retained and transmitted to central storage, then complete data availability is achieved, but storage latency and retrieval time increase
Solution Approach 1:
Edge computing nodes perform preliminary filtering, aggregation, and preprocessing of events before transmission to central storage. By performing these actions in advance at the source, the system reduces the volume of data requiring central storage and retrieval operations, thereby decreasing latency while maintaining data availability for critical events.
Solution Approach 2:
The system implements location-specific data retention policies where edge nodes retain and quickly access frequently accessed or time-sensitive event data locally, while less critical data is archived centrally. This local quality differentiation enables rapid retrieval for important events without the overhead of accessing the entire centralized repository.
3Loss of information
If sensitive information is retained in event data for complete record-keeping, then full event context is preserved, but security risks and compliance issues increase
Solution Approach 1:
The system extracts and removes sensitive information from event data at the edge nodes before transmission to central systems. This extraction process preserves the essential event context and operational information while eliminating personally identifiable information and other sensitive data that would create security and compliance risks in centralized storage.
Solution Approach 2:
Edge computing nodes serve as intermediary processing layers between event sources and central systems. These intermediaries perform local data transformation, filtering, and anonymization, acting as a buffer that protects sensitive information from being exposed in centralized repositories while still enabling effective event monitoring and response.
4Device complexity
If a single database system is used for event storage, then system simplicity is maintained, but scalability and performance under high event volumes are limited
Solution Approach 1:
The system merges multiple specialized database systems into a hybrid architecture that combines the strengths of different database types. Relational databases handle structured event data with complex queries, while NoSQL databases manage high-volume unstructured data and time-series information, creating a unified system that maintains simplicity of use while achieving high scalability and throughput.
Data Source
AI summary
Described in detail herein is an event streaming system. A controller disposed at a facility, can receive an event from computing systems disposed at the facility. The event is received by the controller at a time the event occurs. The controller can stream the event including the set of data. A central system can receive the at least one event streamed by the controller. The central system can include a repository cluster, a relational database, and a NoSQL database. The central system can concurrently route the set of data included in the event to the repository cluster, the relational database and the NoSQL database of the event response system to store the set of data in the repository cluster, the relational database and the NoSQL database.