Distributed Framework for Kafka Data Extraction and Processing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing mechanisms for consuming data from message brokers like Kafka in networked computing environments require substantial custom coding for data extraction and processing, leading to inefficiencies and increased network bandwidth usage.

Innovation Solution

A distributed software framework encapsulates overhead functionality for extracting data from message brokers and writing it to targets, using committer managers, replicator workers, and a ZooKeeper service to facilitate concurrent computing tasks, allowing for custom code plugins to transform data and manage message batches efficiently.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If custom scripts are used to manage data extraction from message brokers, then data processing flexibility is improved, but system complexity and development effort increase

Engineering Contradiction:

Improvedata processing flexibilityVSAvoidsystem complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system divides data extraction functionality into separate plugin modules that can be independently developed and configured. Each plugin handles specific data processing tasks, allowing flexibility without increasing overall system complexity. The framework segments the monolithic custom script approach into modular, reusable components.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent creates a universal framework that can accommodate multiple types of data extraction plugins through a common interface. This multi-functional architecture allows the same base system to handle diverse data processing requirements through pluggable modules, reducing the need for separate custom scripts for each scenario.

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Adaptability or versatility

If custom scripts are handwritten for data extraction, then specific processing requirements are met, but development time and cost increase

Engineering Contradiction:

Improveprocessing requirement fulfillmentVSAvoiddevelopment time

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

The framework performs preliminary actions by providing a pre-built infrastructure for data extraction, including connection management, error handling, and basic processing logic. This head-start reduces the development time required to implement specific processing requirements, as developers only need to add their custom logic on top of the established framework.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system uses template plugins and reusable code structures that can be copied and adapted for different data processing scenarios. Instead of writing custom scripts from scratch for each requirement, developers can replicate and modify existing plugin templates, significantly reducing development time while maintaining flexibility.

Inventive Principle:

Principle #26Copying

3Speed

If data is continuously streamed from message brokers, then real-time processing is achieved, but network bandwidth consumption increases

Engineering Contradiction:

Improvereal-time processing speedVSAvoidnetwork bandwidth consumption

Core Design Contradiction:

SpeedVSLoss of energy

Solution Approach 1:

The framework implements partial data extraction by allowing selective filtering of message batches. Instead of processing every single message in real-time, the system can extract and process only the necessary portions of data streams based on configured criteria, reducing network bandwidth consumption while maintaining real-time processing capabilities for critical data.

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

The system employs periodic batching of data extraction operations, where data is collected in batches at intervals rather than continuously streamed. This periodic approach reduces the peak network bandwidth requirements while still achieving timely processing, as the framework can handle multiple batches sequentially without requiring sustained high bandwidth usage.

Inventive Principle:

Principle #19Periodic action

Data Source

PatentUS10481948B2Data transfer in a collaborative file sharing system

Publication Date: 2019.11.19 BOX INC

AI summary

A system and method for facilitating implementing a computing task (such as selectively transforming and/or writing data retrieved from a message stream of a message broker) via concurrent computing in a networked computing environment. The example method includes using a distributed software framework to encapsulate overhead functionality for facilitating extracting data from a data stream output from the message broker and writing associated data to one or more targets in a networked computing environment; and accommodating custom code as a plugin to one or more modules of the software framework. In a more specific embodiment, the one or more modules include one or more committers. The overhead functionality includes functionality for facilitating selective redistribution of work when a process becomes unreliable; tracking of offsets for message batches in the data stream; automatically assigning work to nodes in the networked computing environment.