Stream Processing Workflow Automation Framework
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Users face challenges in selecting and implementing stream processing frameworks like S4, Storm, and Spark due to their unique APIs and programming styles, requiring significant time and skill to determine the most suitable framework and learn its API for specific use cases, with no automated deployment or common visualization layer across frameworks.
Innovation Solution
A stream processing handling arrangement that provides data handling blocks for users to form workflows, compares workflow objectives with framework capabilities, and automatically selects and constructs applications using software modules specific to the chosen framework, simplifying the selection and implementation process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a user manually investigates and learns multiple stream processing framework APIs, then the user can determine the most suitable framework for specific use cases, but this process requires significant time and skill investment
Solution Approach 1:
The patent introduces an intermediary system that acts as a mediator between the user and multiple stream processing frameworks. This system automatically compares workflow requirements with framework capabilities, eliminating the need for users to manually investigate each framework's API and characteristics, thus resolving the contradiction between framework adaptability and time investment
Solution Approach 2:
The patent creates a virtual representation or model of multiple stream processing frameworks, allowing users to interact with a simplified interface that replicates the essential characteristics of each framework without requiring direct engagement with their complex APIs, thereby reducing the time and skill barrier
2Ease of manufacture
If a user learns the API of a chosen stream processing framework, then the user can implement applications in that framework, but this requires considerable skills by the user
Solution Approach 1:
The system serves as an intermediary that translates high-level workflow requirements into framework-specific code, eliminating the need for users to learn complex framework APIs. The intermediary handles the complexity of API interactions while users work with a simplified, framework-agnostic interface
Solution Approach 2:
The patent creates a universal interface that works across multiple stream processing frameworks, allowing users to implement applications without needing to learn framework-specific APIs. The system provides multi-functionality by supporting various frameworks through a single unified interface, reducing the skill barrier
Data Source
AI summary
A stream processing handling arrangement provides a number of data handling blocks, selectable for forming a work flow. The blocks comprise data processing blocks. The arrangement receives selections of data handling blocks, block parameter settings and workflow property objectives, and compares the objectives with corresponding stream processing framework capabilities. The arrangement determines a stream processing framework for the work flow, provides a number software module groups, each corresponding to a different framework and comprising software modules corresponding to the data handling blocks, and chooses a group corresponding to the determined stream processing framework. The arrangement selects, in this group, software modules corresponding to the selected blocks and constructs an application using the selected software modules.