Real-Time Data Processing Engine for Non-Developer Users
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data processing tools are either too application-specific for non-developer users or too generic, lacking simple controls for sophisticated data processing and analytics, and often require developer support for handling real-time, high-frequency data streams.
Innovation Solution
A web-based graphical user interface allows non-developer users to create, edit, and deploy complex processing rules on streaming or static data feeds, using a scalable data processing engine that interprets user instructions in real-time and generates desired outputs, such as alerts, without requiring developer support.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If application-specific rule engines are used to provide simple controls for non-developer users, then ease of operation is improved, but adaptability deteriorates because they cannot provide sophisticated data processing and analytics
Solution Approach 1:
The system employs a universal rule engine that can handle both simple operations for non-developer users and sophisticated data processing for complex analytics. The engine is designed to be multi-functional, supporting various data sources, multiple analysis types, and diverse output formats within a single platform, thereby resolving the contradiction between ease of operation and adaptability
2Ease of operation
If generic rule engines with simple user controls are used, then ease of operation is improved, but manufacturing precision deteriorates because they lack sophisticated data processing capabilities
Solution Approach 1:
The system segments the rule engine into modular components that can be selectively activated. Non-developer users interact with a simplified interface for basic operations, while the backend maintains sophisticated processing capabilities that can be engaged when needed, thus preserving both ease of operation and data processing precision
3Productivity
If engines handling large sets of real-time high-frequency streaming data are used, then productivity is improved, but device complexity worsens requiring highly trained developers to operate
Solution Approach 1:
The system introduces an intermediary layer between the complex real-time data processing engine and the user interface. This intermediary handles the complexity of high-frequency streaming data processing while presenting a simplified interface to users, thereby maintaining high productivity without requiring highly trained developers to operate the system
4Reliability
If multi-layered deployment processes are used for each rule added, then reliability is improved through thorough testing, but loss of time worsens due to non-real-time deployment
Solution Approach 1:
The system performs preliminary actions by pre-compiling and validating rules during development and testing phases before deployment. This preliminary validation ensures reliability while enabling faster deployment, as the multi-layered testing is already completed before the rule is activated in production, reducing the actual deployment time
Data Source
AI summary
A computer-implemented method is provided for permitting a user to manipulate data feeds via a graphical user interface. The method includes receiving, by a computing device, the data feeds over a communications network in real time from corresponding data sources and receiving from the user, by the computing device, via the graphical user interface an execution plan comprising a list of one or more actions to be performed on the data feeds. The method also includes sequentially executing, by the computing device, each of the actions in the execution plan on the real-time data feeds to generate one or more manipulated data feeds. The method further includes displaying, by the computing device, the manipulated data feeds to the user in a user-defined format via the graphical user interface.