Stream Processing System for Real-Time Keyed Data Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems face latency and inefficiencies in processing and analyzing data due to multiple disparate systems re-formatting data multiple times, leading to delayed interactions and inability to process data in real-time or near real-time, which renders dependent actions obsolete.
Innovation Solution
A system that integrates data collection, detection, and action into a single execution system, allowing for real-time or near real-time processing of data streams without storing data to disk, using a collect-detect-act (CDA) environment with parameterized applications to handle batch and real-time data streams, reducing latency and memory consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple disparate systems process and re-format data records sequentially, then each system can process data of its particular type, but latency increases and real-time processing capability is lost
Solution Approach 1:
The patent merges multiple disparate data processing systems into a single integrated system that can handle multiple data types simultaneously. The system consolidates collection, validation, formatting, and processing functions into one unified architecture, eliminating the sequential handoff between separate systems and their associated latency.
Solution Approach 2:
The integrated system is designed with multi-functionality to process various types of data records through a single platform. It incorporates multiple processing engines and adapters that can handle different data formats and types within the same system, replacing the need for multiple specialized disparate systems.
2Adaptability or versatility
If each system performs its own data formatting and reformatting operations, then data can be processed by each system, but latency accumulates with each formatting operation
Solution Approach 1:
The system performs preliminary data validation and standardization at the point of data intake, before the data enters the processing pipeline. By pre-formatting and validating data records upfront, the system eliminates the need for repeated formatting operations at each subsequent processing stage, reducing cumulative latency.
Solution Approach 2:
The integrated system enforces a standardized internal data format and structure across all processing components. Data is transformed into a homogeneous format once upon entry, ensuring consistency throughout the system without requiring repeated formatting operations between different processing stages.
3Productivity
If data is processed and analyzed with delays, then processing can be performed, but dependent actions become obsolete when the current situation changes
Solution Approach 1:
The system implements continuous real-time processing of data streams without batch delays or intermediate storage. Data flows continuously through the processing pipeline from collection to analysis and action generation, ensuring that processing actions are always based on current data states and remain timely and relevant.
4Quantity of substance
If conventional systems process data in batch or with storage to disk, then data can be processed, but real-time or near real-time processing capability is lost
Solution Approach 1:
The system replaces mechanical disk storage and batch processing mechanisms with in-memory processing and stream computing architectures. By utilizing memory-based data structures and continuous processing pipelines instead of disk-based batch operations, the system achieves real-time processing speeds while handling large data volumes.
Data Source
Figure 1
Figure 2A~2B
Figure 2C
AI summary
A method is described for processing keyed data items that are each associated with a value of a key, the keyed data items being from a plurality of distinct data streams, the processing including collecting the keyed data items, determining, based on contents of at least one of the keyed data items, satisfaction of one or more specified conditions for execution of one or more actions and causing execution of at least one of the one or more actions responsive to the determining.