Stream Processing State Migration for Dynamic Scaling

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing stream processing systems face challenges in dynamically adjusting computing resources at runtime to match changing workloads, leading to inefficiencies and potential latency issues, especially when handling real-time data processing applications like video stream monitoring.

Innovation Solution

A method and system that dynamically adjust the number of consumer instances by determining an adjustment policy for migrating intermediate processing states between existing and new instances, allowing for real-time scaling of computing resources without interrupting data processing, using a consistent hashing algorithm to minimize migration overhead.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If the number of consumer instances is increased to handle higher workload, then processing capacity is improved, but system complexity and resource management overhead increase

Engineering Contradiction:

Improveprocessing capacityVSAvoidsystem complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system dynamically adjusts the number of consumer instances based on real-time workload monitoring. When workload exceeds threshold, new instances are created; when workload decreases, instances are terminated. This dynamic adaptation resolves the contradiction by making the system flexible rather than static, allowing processing capacity to scale with demand without permanent complexity overhead.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system implements feedback mechanisms through workload monitoring and performance metrics collection. This feedback loop enables automatic adjustment of consumer instance numbers, where performance data triggers resource scaling decisions. The feedback principle resolves the contradiction by using empirical data to optimize resource allocation, preventing both over-provisioning and under-provisioning.

Inventive Principle:

Principle #23Feedback

2Adaptability or versatility

If computing resources are dynamically adjusted at runtime, then adaptability to workload changes is improved, but data processing continuity may be affected

Engineering Contradiction:

Improveadaptability to workloadVSAvoiddata processing continuity

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The system performs preliminary actions by pre-warming consumer instances before they are needed and pre-distributing data partitions. When scaling up, instances are initialized in advance rather than waiting for demand. Data is pre-partitioned and staged for distribution, reducing the impact of scaling operations on processing continuity and maintaining reliability during adaptation.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system maintains continuity of useful action by ensuring data processing does not cease during resource adjustment. Consumer instances continue processing their assigned data partitions while new instances are being prepared and integrated. The data flow and processing pipeline remain active throughout the scaling operation, preventing interruptions and maintaining reliability while achieving adaptability.

Inventive Principle:

Principle #20Continuity of useful action

3Productivity

If consumer instances are added or removed, then resource efficiency is improved, but state migration overhead increases

Engineering Contradiction:

Improveresource efficiencyVSAvoidstate migration time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The system segments the state migration process into independent, parallelizable units. Data is divided into partitions that can be migrated independently between instances. This segmentation allows multiple state transfers to occur simultaneously rather than sequentially, reducing total migration time while enabling flexible resource allocation for improved efficiency.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system changes parameters such as batch size, parallelism degree, and migration timing based on system conditions. Migration parameters are dynamically adjusted - larger batches when bandwidth is available, more parallel streams when target instances are ready. This parameter optimization minimizes migration overhead while maintaining resource efficiency, resolving the contradiction between productivity and time loss.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS11023274B2Method and system for processing data

Publication Date: 2021.06.01 EMC IP HLDG CO LLC

AI summary

A method for processing data includes receiving an adjustment request for adjusting a number of consumer instances from a first number to a second number, and determining a migration overhead for adjusting a first distribution of states associated with the first number of consumer instances to a second distribution of the states associated with the second number of consumer instances, wherein the states are intermediate results of processing the data and the migration overhead includes a latency and a bandwidth shortage incurred for migrating the states. Based on the determined migration overhead, the states are migrated between the first number of consumer instances and the second number of consumer instances, and thereafter the data is processed based on the second distribution of the states at the second number of consumer instances.