Live Partition Resegmentation in Stream Processing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional stream-processing platforms, such as Apache Kafka, face performance degradation and service disruptions when trying to move a partition from one broker to another without replicating all data, as the replication process overloads the old physical location, causing request drops.

Innovation Solution

Implementing live resegmentation in distributed stream-processing platforms, where a partition is moved from one broker to another without replicating existing messages, by redirecting write requests to the new broker and allowing the old broker to handle read requests for messages before the last offset, and eventually tearing down the old broker after a retention period.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If data replication is performed when moving a partition from one broker to another, then data consistency is maintained, but the old broker becomes overloaded and requests are dropped

Engineering Contradiction:

Improvedata consistencyVSAvoidrequest processing capacity

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The system performs preliminary actions by creating a new broker instance and allocating the partition to it before the old broker is taken offline. This allows the partition to be moved without replicating data during the transition, avoiding overload on the old broker while maintaining data consistency through the offset-based approach.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The partition movement process is segmented into distinct phases: allocation to new broker, offset tracking, selective message forwarding for uncommitted messages, and eventual cleanup. This segmentation allows the system to maintain reliability while avoiding the performance degradation caused by full data replication.

Inventive Principle:

Principle #1Segmentation

2Reliability

If all messages are replicated from old broker to new broker, then data consistency is ensured, but the replication process increases load on the partition

Engineering Contradiction:

Improvedata consistencyVSAvoidpartition load

Core Design Contradiction:

ReliabilityVSStress or pressure

Solution Approach 1:

Instead of replicating all messages, the system performs partial replication by only forwarding uncommitted messages (those with offsets greater than the last committed offset) from the old broker to the new broker. This partial action maintains data consistency for committed messages while avoiding the excessive load of full replication.

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

The new broker independently manages its own partition data by receiving only the necessary uncommitted messages from the old broker, rather than relying on full replication. This self-service approach reduces the burden on the old broker while ensuring the new broker has all necessary data.

Inventive Principle:

Principle #25Self-service

3Reliability

If the old broker continues to serve read requests during partition move, then service continuity is maintained, but the broker cannot be torn down for capacity management

Engineering Contradiction:

Improveservice continuityVSAvoidcapacity management flexibility

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

The system performs preliminary allocation of the partition to the new broker before the old broker is torn down. This preliminary action ensures service continuity during the transition while enabling the old broker to be reclaimed for capacity management purposes afterward.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The new broker acts as an intermediary that takes over the partition before the old broker is decommissioned. This intermediary approach maintains service continuity while enabling flexible capacity management by allowing the old broker resources to be reallocated.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS10979363B2Live resegmenting of partitions in distributed stream-processing platforms

Publication Date: 2021.04.13 ORACLE INT CORP

AI summary

Techniques for resegmenting a partition in a distributed stream-processing platform are provided. The techniques include receiving a trigger to move a partition of the distributed stream-processing platform from a first broker on a first set of physical resources to a second broker on a second a set of physical resources. In response to the trigger, the partition is allocated on the second broker, and the first broker is configured to redirect, to the second broker, requests for new messages after a last offset in the partition without replicating older messages before the last offset to the second broker. Idempotent produce metadata for the partition from the first broker is then merged into the second broker. Finally, metadata for processing requests for the partition is updated to include the second broker.