Big Data Ingestion Framework for Balanced Streaming Throughput

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current technologies lack a comprehensive framework for integrating, processing, and analyzing large volumes of big data from diverse sources, including File-based and Streaming Datasets, while addressing issues such as workload imbalance, resource underutilization, security concerns, and failure resilience.

Innovation Solution

A data ingestion and processing framework that facilitates ingestion and analysis of big data from various sources, enabling real-time processing, fault tolerance, customizable throughput, and parallelism, with features like schema validation, multiple sinks, and resilient data handling.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If more resources are added to handle large data streams, then processing capacity is improved, but workload imbalance and resource underutilization worsen

Engineering Contradiction:

Improveprocessing capacityVSAvoidworkload balance

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The system segments data streams into multiple channels based on schema types and processing requirements. Different workers handle different schema categories (e.g., JSON, XML, Avro) separately, preventing workload imbalance while maintaining high processing capacity through parallelization.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The framework dynamically adjusts resource allocation based on real-time data characteristics and processing needs. Workers can be dynamically added, removed, or reassigned based on current workload patterns, ensuring optimal resource utilization without fixed rigid assignments.

Inventive Principle:

Principle #15Dynamics

2Adaptability or versatility

If data is stored in raw formats for flexibility, then adaptability is improved, but storage requirements and processing time worsen

Engineering Contradiction:

Improvedata format flexibilityVSAvoidstorage requirements

Core Design Contradiction:

Adaptability or versatilityVSQuantity of substance

Solution Approach 1:

The system applies different compression and formatting strategies to different data types and schemas. Critical data is compressed more aggressively while less important data uses lighter compression, optimizing storage efficiency without sacrificing necessary adaptability for diverse data formats.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The framework automatically transforms data parameters (compression level, encoding format, partitioning strategy) based on schema detection and data characteristics. This allows the system to adapt storage parameters dynamically to minimize storage requirements while maintaining flexibility.

Inventive Principle:

Principle #35Parameter changes

3Speed

If real-time processing is implemented for streaming data, then responsiveness is improved, but system complexity and failure risk worsen

Engineering Contradiction:

Improveprocessing speedVSAvoidsystem complexity

Core Design Contradiction:

SpeedVSDevice complexity

Solution Approach 1:

The framework introduces a mediation layer between data sources and processing workers that handles schema validation, data transformation, and error handling centrally. This intermediary layer simplifies individual worker complexity while maintaining real-time processing capabilities through standardized interfaces.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

Schema validation and data preparation are performed in advance before data reaches processing workers. Pre-defined schemas and transformation rules are applied beforehand, reducing the complexity of real-time processing decisions and enabling faster execution.

Inventive Principle:

Principle #10Preliminary action

4Manufacturing precision

If comprehensive schema validation is applied to all data, then data quality is improved, but processing throughput and tolerance worsen

Engineering Contradiction:

Improvedata qualityVSAvoidprocessing throughput

Core Design Contradiction:

Manufacturing precisionVSProductivity

Solution Approach 1:

The system applies comprehensive schema validation selectively based on data importance, schema complexity, and processing requirements. Critical data streams receive full validation while less critical streams receive reduced validation, maintaining data quality where needed while preserving overall processing throughput.

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

Common data quality issues are prevented in advance through pre-defined schemas and validation rules that reject malformed data before processing. This preliminary validation reduces the burden on downstream processing stages and maintains throughput by preventing errors from propagating.

Inventive Principle:

Principle #9Preliminary anti-action

Data Source

PatentUS12524424B2System and method of data ingestion and processing framework

Publication Date: 2026.01.13 JIO PLATFORMS LTD

AI summary

The present disclosure provides for a system (100) and method (250) that facilitates a robust and effective solution to an entity or an organization by enabling ingestion and processing of a set of data packets corresponding to large streaming and batched data in a big data eco-system that may be fast and may provide optimal throughput, rate control, throttle and embedded fault tolerance. The system (100) is equipped with a data ingestion module (110) to receive the set of data packets and extract a set of attributes pertaining to the nature of data from the set of data packets received and then poll a plurality of second computing devices (104) based on the set of attributes extracted; and, upon establishing polling, store the set of data packets in the plurality of second computing devices (104).