Dynamic Data Router for Real-Time Event Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data processing systems struggle to process data in real-time, leading to latency issues, performance problems, and data integrity challenges, particularly in IT event analysis.
Innovation Solution
A method and system for routing and processing real-time information event technology data using dynamic configuration settings, which involves receiving a configuration file, extracting data from an external source, applying transformations, and routing data to appropriate applications or analytics sinks based on the configuration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If conventional batch processing is used for IT event data, then data processing can be performed with simpler system requirements, but real-time processing capability and latency performance deteriorate
Solution Approach 1:
The patent segments the data processing system into multiple independent components: event sources, event processors, event routers, and data sinks. Each component handles specific tasks independently, enabling real-time processing while maintaining manageable system complexity through modular architecture.
Solution Approach 2:
The patent implements dynamic configuration files that allow the system to adapt its processing behavior in real-time based on changing requirements. The configuration files enable dynamic routing rules, transformation parameters, and processor assignments to be modified without system reconfiguration, supporting real-time processing demands.
2Loss of time
If real-time data processing is implemented, then latency is reduced and data freshness is improved, but system resource consumption and complexity increase
Solution Approach 1:
The patent applies partial processing by allowing different event processors to handle different subsets of events based on configuration files. Not all events require the same level of processing intensity, and the system can adjust processing depth dynamically, reducing overall resource consumption while maintaining real-time latency performance for critical events.
Solution Approach 2:
The patent uses configuration files as reusable templates that define processing patterns, routing rules, and transformation logic. These configuration files can be copied and adapted for different event types and destinations, reducing the need to create new processing logic from scratch and lowering system resource requirements.
3Adaptability or versatility
If dynamic configuration settings are used for data routing, then routing flexibility and adaptability are improved, but configuration management complexity increases
Solution Approach 1:
The patent implements universal configuration files that can serve multiple purposes: defining routing rules, specifying transformation parameters, identifying target data sinks, and configuring processor behavior. This multi-functionality reduces the need for separate configuration mechanisms and simplifies overall configuration management while maintaining high routing flexibility.
Solution Approach 2:
The system enables self-service configuration through automated configuration file generation and validation. The event router and processors can automatically validate configuration syntax, resolve dependencies, and apply configurations without manual intervention, reducing configuration management complexity while preserving adaptability.
4Manufacturing precision
If comprehensive data transformations are applied to all fields, then data quality and consistency are improved, but processing time and computational resources increase
Solution Approach 1:
The patent applies local quality by allowing different transformation rules to be applied to different fields within events based on configuration files. Critical fields receive comprehensive validation and transformation, while less critical fields receive minimal processing. This selective approach maintains data quality for essential information while preserving processing throughput.
Solution Approach 2:
The patent uses configuration files to dynamically change processing parameters such as transformation depth, validation strictness, and field selection criteria. These parameters can be adjusted based on event type, destination, and system load, enabling the system to maintain data quality while optimizing processing throughput for different scenarios.
Data Source
AI summary
Computer implemented system and method for routing and processing real-time information event technology data. The method including: receiving a configuration file associated with an external data source; extracting a stream of data from the external data source, the stream of data including one or more data objects, wherein each of the one or more data objects includes a plurality of fields; applying, through a processing platform, a transformation to the one or more data objects, wherein applying the transformation includes applying the transformation to a first field of the plurality of fields for each of the one or more data objects; and routing the one or more data objects to a separate application, storage system, or processing analytics data sink based on the configuration file