Iterative Pipeline for Parallel Data Partitioning

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing large-scale data processing systems face challenges in maximizing throughput for network-intensive processing jobs, leading to system overload, underutilization of resources, and high costs due to peak usage allocation and sequential processing limitations.

Innovation Solution

Implementing an iterative pipelining approach where records are partitioned and processed in parallel through multiple stages, allowing concurrent processing and merging of partitions, optimized for cloud-based systems like Hadoop and MapReduce, to control request rates and enhance overall throughput.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If cloud services are provisioned for peak capacity to handle large-scale processing jobs, then system reliability is improved, but resource utilization deteriorates as services remain underutilized for most of the day

Engineering Contradiction:

Improvesystem reliabilityVSAvoidresource utilization

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The system dynamically adjusts the number of processing nodes and resource allocation based on real-time workload conditions. During peak periods, additional nodes are activated to handle large-scale jobs reliably; during off-peak periods, nodes are deactivated or placed in low-power states, optimizing resource utilization while maintaining reliability when needed

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

Instead of provisioning for full peak capacity continuously, the system activates only the necessary portion of resources based on current demand. This partial action approach ensures reliability during peak loads while avoiding the waste of keeping excessive resources idle during off-peak periods

Inventive Principle:

Principle #16Partial or excessive action

2Stability of the object's composition

If sequential processing is used in existing data processing systems, then service stability is improved, but throughput deteriorates as only one service may be utilized at a time

Engineering Contradiction:

Improveservice stabilityVSAvoidthroughput

Core Design Contradiction:

Stability of the object's compositionVSProductivity

Solution Approach 1:

The processing system is segmented into independent, modular services that can operate concurrently. Each service handles specific processing tasks independently, allowing multiple services to run in parallel without interfering with each other, thereby increasing throughput while maintaining individual service stability

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system transitions from single-dimensional sequential processing to multi-dimensional parallel processing by introducing additional processing dimensions through iterative pipelines. Multiple partitions of data can be processed simultaneously through different stages, effectively adding temporal and spatial dimensions to the processing architecture

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

3Quantity of substance

If large input size is processed in existing systems, then data completeness is improved, but system overload occurs generating a large number of map tasks

Engineering Contradiction:

Improvedata completenessVSAvoidsystem overload

Core Design Contradiction:

Quantity of substanceVSObject-affected harmful factors

Solution Approach 1:

Large input datasets are automatically segmented into multiple manageable partitions before processing. Each partition is handled by dedicated processing nodes in the iterative pipeline, preventing any single node from being overwhelmed while ensuring complete data processing through coordinated merging of partition results

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system introduces intermediary buffering and coordination layers between data input and processing stages. These intermediaries manage the flow of large datasets, regulating input rates and distributing workload evenly across processing nodes, thereby preventing system overload while maintaining data completeness

Inventive Principle:

Principle #24Intermediary (Mediator)

4Productivity

If iterative pipelining with parallel processing is implemented, then throughput is improved, but system complexity increases

Engineering Contradiction:

ImprovethroughputVSAvoidsystem complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The iterative pipeline framework is designed as a universal, reusable architecture that can handle various processing workloads through configuration rather than custom implementation. Standardized components and conventions reduce the complexity burden despite the parallel processing capabilities, allowing the same framework to serve multiple functions across different data processing scenarios

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS10326824B2Method and system for iterative pipeline

Publication Date: 2019.06.18 R2 SOLUTIONS LLC

AI summary

Method, system, and programs for data processing. In one example, a record is received. The record is separated into multiple partitions. Each partition is submitted to an iterative pipeline, wherein the iterative pipeline comprises two or more processing stages that are run in parallel and one of the multiple partitions passes through one of the two or more processing stages in one iteration. The multiple partitions are passed through two or more processing stages. The partitions that have passed through each of the two or more processing stages are merged.