Event Topic Checkpointing for Ingestion Reliability

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional event ingestion management techniques often result in event loss during high-volume ingestion from event topics, leading to incomplete processing and requiring re-ingestion of all events to correct unconsumed ones.

Innovation Solution

Implementing event consumption checkpointing via a durable storage repository, where events are polled, checkpointed in multiple stages, and processed with asynchronous evaluation to identify and reintroduce unconsumed events at optimal injection points.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If conventional event ingestion management techniques are used, then event processing can be initiated, but event loss occurs during high-volume ingestion from event topics

Engineering Contradiction:

Improveevent ingestion rateVSAvoidevent consumption accuracy

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The system performs preliminary checkpointing of event consumption state to durable storage before event processing completes. This preliminary action records the last successfully processed event offset, enabling the system to resume from the correct position after failures without losing events or requiring re-ingestion of all events.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent introduces an intermediary checkpointing mechanism that mediates between the event consumer and the event topic. This intermediary layer maintains a durable record of consumption state, acting as a buffer that prevents direct loss of event position information during high-volume ingestion or system failures.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If conventional event ingestion techniques are used, then events can be processed, but unconsumed events require re-ingestion of the complete volume of events

Engineering Contradiction:

Improveevent processing throughputVSAvoidtime for re-ingesting events

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

Inventive Principle:

Principle #10Preliminary action

3Reliability

If checkpointing in multiple stages is implemented, then event consumption accuracy is improved, but system complexity increases

Engineering Contradiction:

Improveevent consumption accuracyVSAvoidcheckpointing system complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent segments the checkpointing process into multiple stages: initial checkpoint creation, intermediate updates during processing, and final commitment. Each stage handles a specific aspect of checkpoint management, allowing the system to maintain high accuracy while organizing complexity into manageable, modular components.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS12182638B2Method and system for event topic checkpointing

Publication Date: 2024.12.31 JPMORGAN CHASE BANK NA

AI summary

A method for providing ingestion checkpointing from an event topic is disclosed. The method includes polling, via an event consumer, a message broker to initiate consumption of events from the event topic, the event topic relating to event storage; checkpointing, based on a result of the polling, the consumption of the events by writing corresponding data in multiple stages to a checkpointing durable store; inserting an event pending status for each of the events in an events time bucket table; committing, via the event consumer, the consumption of the events; initiating event processing for the events; and updating the events time bucket table with an event complete status when consumption of the events are completed.