Event Translator Schema for Streaming Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale data processing systems face difficulties in efficiently categorizing and analyzing vast amounts of log data due to disparate data formats and resource constraints, leading to slow search results and computational inefficiencies.
Innovation Solution
An event monitoring system that employs an event translator to classify and transform event data into a common schema, using filters like regular expression filters, and computes engines to process events in real-time, optimizing data processing and analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in large-scale data stores with millions to billions of entries, then data storage capacity is improved, but search speed and computational efficiency deteriorate
Solution Approach 1:
The patent segments the monolithic data store into multiple specialized indexes organized by event type, user, time range, and other dimensions. Each index contains only the subset of events relevant to its classification, allowing parallel queries to operate on smaller, more manageable data partitions rather than scanning the entire billion-entry data store.
Solution Approach 2:
The system performs preliminary actions by pre-processing incoming events and organizing them into multiple indexes in real-time as they arrive. This pre-organization into categorized indexes occurs before any search operations, enabling rapid retrieval without requiring full data store scans when queries are executed.
2Loss of information
If comprehensive log data is collected from various sources, then data completeness is improved, but data format disparity and processing complexity increase
Solution Approach 1:
The patent introduces an event translator as an intermediary component that sits between diverse data sources and the indexing system. This translator receives events in various formats from different sources, automatically translates and normalizes them into a unified event schema, and then routes them to appropriate indexes. This mediator eliminates the need for complex custom processing logic for each data source while maintaining complete data collection.
3Loss of time
If real-time event processing is implemented, then analysis timeliness is improved, but computational resource consumption increases
Solution Approach 1:
The system segments the processing workload by creating specialized indexes for different event types and dimensions. Each index can be independently queried and processed, allowing parallel computation across multiple indexes rather than sequential processing of a single large data set. This segmentation reduces the computational resources required for each individual query while maintaining real-time processing capabilities.
Solution Approach 2:
The patent implements partial action by allowing users to query only specific subsets of indexes relevant to their analysis needs, rather than processing or scanning the entire event data stream. For example, a query for login events only accesses the login-specific index, processing a fraction of the total events while delivering real-time results with minimal resource consumption.
Data Source
AI summary
Disclosed are various embodiments for filters and event schema for categorizing and processing streaming event data. An event may be generated by a service that describes user interaction a client application executable on a client device. The event may be received as a data structure in a first format. A filter, such as a regular expression filter, may be applied to the data structure to identify an event type for the event. The data structure may be converted from the first format to a second format in accordance with a common event schema able to be interpreted by virtual compute engines.