Stream Processing Component Duplication for Resource Constraint Satisfaction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed stream processing systems often face instability and data processing faults due to computing resource mismatches between stream processing components and available computing nodes, leading to degraded system performance.
Innovation Solution
A method that calculates the computing resources required for each stream processing component, duplicates components as needed to meet resource constraints, and allocates them to nodes with sufficient resources, using data allocation and resource allocation policies to manage data flow and resource distribution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If stream processing components are allocated to computing nodes without resource verification, then allocation speed is improved, but system stability deteriorates due to resource mismatches
Solution Approach 1:
The patent performs preliminary resource verification and component duplication before allocation. The system calculates required computing resources for each component, verifies availability on target nodes, and creates duplicate components with divided resource requirements in advance, ensuring that when allocation occurs, resource mismatches are prevented.
Solution Approach 2:
The patent segments a single stream processing component into multiple duplicate components when resource constraints exist. By dividing the original component's processing load across multiple instances with reduced individual resource requirements, the system can allocate to nodes with limited resources while maintaining overall processing capability.
2Adaptability or versatility
If stream processing components are duplicated to meet resource constraints, then resource utilization is improved, but system complexity increases
Solution Approach 1:
The patent changes the resource allocation parameters by creating components with divided resource requirements. When an original component requires more resources than available, the system creates duplicate components where each instance requires a portion of the original resources, allowing flexible adaptation to different node capacities.
Solution Approach 2:
The patent creates duplicate copies of stream processing components to match available computing resources. By copying components and dividing their resource requirements, the system can utilize nodes with varying resource capacities, improving overall resource utilization across the distributed system.
3Reliability
If computing resources are divided among duplicate components, then resource constraint satisfaction is improved, but data processing overhead increases
Solution Approach 1:
The patent merges multiple duplicate components back together in terms of their collective processing output. While individual duplicate components have reduced resource requirements, they work in parallel to process the same data stream, and their combined output satisfies the original processing requirements without excessive overhead.
Data Source
AI summary
The invention discloses a stream processing method, which includes: receiving a first stream processing task including one or more stream processing components, a data input and output relationship of the stream processing component, and an identifier of a stream data source; calculating a computing resource required by each stream processing component; and if the first stream processing task includes a first stream processing component which does not meet a preset constraint condition, duplicating the first stream processing component to obtain at least one second stream processing component, so that the data may be allocated to the first stream processing component and the second stream processing component(s), which reduce a probability that system instability and a data processing fault are caused.