Mainframe Data Distribution via Kafka Event Platform

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current methods for data integration between mainframes and distributed environments are limited, requiring complex coordination and expensive solutions, making it difficult to efficiently distribute data from mainframes to multiple subscribers.

Innovation Solution

A system and method that decouples data distribution from mainframes to distributed platforms by using a data publisher to curate and publish data feeds to event platforms like Apache Kafka, allowing for efficient distribution and access by multiple consumers without requiring coordination between the producer and consumer.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If traditional data integration methods (FTP, NFS, NDM, MQ) are used to distribute data from mainframe to distributed platforms, then data distribution can be achieved, but the system complexity increases and coordination overhead is required between producer and consumer

Engineering Contradiction:

Improvedata distribution reliabilityVSAvoidintegration system complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent introduces an event platform (Kafka) as an intermediary between the mainframe data producer and distributed consumers. The mainframe publishes data to Kafka topics, and consumers subscribe to these topics independently. This intermediary eliminates the need for direct coordination between producers and consumers, reducing system complexity while maintaining reliable data distribution through the event platform's built-in messaging capabilities.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Adaptability or versatility

If coordinated data distribution is used where each subscriber requires setup between data provider and consumer, then data can be distributed to multiple endpoints, but duplicative steps and processing cycles increase

Engineering Contradiction:

Improvemulti-subscriber data distributionVSAvoidcoordination time

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

The patent segments the data distribution system into independent publishers and subscribers that communicate through standardized Kafka topics. Each subscriber can independently subscribe to topics of interest without requiring coordination with the data provider. This segmentation allows multiple endpoints to receive data simultaneously without duplicative coordination steps, reducing time loss while maintaining adaptability to multiple subscribers.

Inventive Principle:

Principle #1Segmentation

3Reliability

If expensive enterprise-level solution stacks (NDM/Connect: Direct) are used for mainframe data integration, then robust data distribution is achieved, but cost increases significantly

Engineering Contradiction:

Improvedata distribution robustnessVSAvoidintegration cost

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The patent replaces expensive enterprise-level integration stacks with open-source, cost-effective alternatives. The mainframe connects to Apache Kafka using standard Java libraries that are already available in the runtime environment, eliminating the need for costly proprietary software like NDM or Connect: Direct. This approach maintains robust data distribution through Kafka's reliable messaging while significantly reducing integration costs.

Inventive Principle:

Principle #27Cheap short-living objects (Disposable)

4Adaptability or versatility

If obtuse 3rd party libraries (C/FLOW, Rocket) are required for Java-based distributed developers, then mainframe data access is enabled, but development complexity and learning curve increase

Engineering Contradiction:

Improvemainframe data accessibilityVSAvoiddeveloper ease of use

Core Design Contradiction:

Adaptability or versatilityVSEase of operation

Solution Approach 1:

The patent leverages the universality of Apache Kafka and standard Java libraries that are already part of the mainframe's runtime environment. Java-based distributed developers can access mainframe data using familiar Kafka consumer libraries and standard Java APIs, eliminating the need for specialized 3rd party libraries like C/FLOW or Rocket. This universal approach maintains mainframe data accessibility while significantly improving ease of operation for developers.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS20240338247A1Systems and methods for facilitating decoupled distribution from a mainframe to distributed platforms

Publication Date: 2024.10.10 THE BANK OF NEW YORK MELLON

AI summary

Systems and methods for facilitating decoupled distribution from a mainframe to one or more distributed platforms curate a data feed from the mainframe; read the data feed; generate an input file for distribution to an event platform topic representing the data feed; in which the event platform topic is hosted on an event platform; write the input file to the event platform topic; and broadcast the data feed to one or more distributed consumers upon request from a given distributed consumer.