Pravega Storage Primitive for Exactly Once Streaming Data Writes

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data processing technologies for streaming applications are complex and costly, particularly in Lambda architectures, which are expensive to develop and manage, and may not be suitable for continuous processing needs like IoT applications, where there is a need for a simpler approach to handle late, out-of-order data and ensure accurate, real-time processing without data duplication or loss.

Innovation Solution

The introduction of Pravega, a new storage primitive that allows for efficient processing of unbounded data by using a durable, append-only sequence of bytes, combined with stream processing engines like Apache Flink, enabling exactly once semantics, auto-scaling, and efficient storage with tiered storage to handle high throughput and low latency, ensuring data is processed accurately and reliably without duplication or loss.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Manufacturing precision

If Lambda architecture is used for streaming data processing, then data processing accuracy is improved, but system complexity and development cost increase

Engineering Contradiction:

Improvedata processing accuracyVSAvoidsystem complexity

Core Design Contradiction:

Manufacturing precisionVSDevice complexity

Solution Approach 1:

The patent merges the speed layer and batch layer into a unified stream processing architecture using Pravega as the common storage backend. This eliminates the need for separate Lambda architecture components while maintaining both real-time processing capabilities and accurate data analysis, thereby reducing system complexity without sacrificing processing accuracy.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

Pravega serves as a universal storage primitive that supports both real-time streaming processing and batch processing workloads. This multi-functional storage system replaces the need for separate storage systems in Lambda architecture, simplifying the overall system while maintaining data processing accuracy across different processing modes.

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Productivity

If Lambda architecture is deployed in production, then data processing capability is improved, but deployment and management cost increase

Engineering Contradiction:

Improvedata processing capabilityVSAvoiddeployment and management cost

Core Design Contradiction:

ProductivityVSEase of manufacture

Solution Approach 1:

The patent combines multiple middleware tools and processing layers into a single streamlined system using Pravega. This consolidation reduces the number of components that need to be deployed and managed, thereby maintaining data processing capability while significantly reducing deployment and management costs.

Inventive Principle:

Principle #5Merging (Combining)

3Manufacturing precision

If batch style processing is used, then data analysis accuracy is improved, but processing speed decreases

Engineering Contradiction:

Improvedata analysis accuracyVSAvoidprocessing speed

Core Design Contradiction:

Manufacturing precisionVSSpeed

Solution Approach 1:

The patent implements a dynamic processing architecture where data can be processed in real-time streams when speed is critical, and in batch mode when accuracy is paramount. The system can dynamically adjust processing modes based on requirements, maintaining both high processing speed and accurate data analysis as needed.

Inventive Principle:

Principle #15Dynamics

4Speed

If real-time processing is implemented, then processing speed is improved, but data accuracy may be reduced

Engineering Contradiction:

Improveprocessing speedVSAvoiddata accuracy

Core Design Contradiction:

SpeedVSManufacturing precision

Solution Approach 1:

The system dynamically adjusts processing modes based on requirements. For time-critical operations, real-time stream processing provides high speed with adequate accuracy. For operations requiring high precision, the system can switch to batch processing modes, ensuring data accuracy is maintained when needed while preserving processing speed when required.

Inventive Principle:

Principle #15Dynamics

5Productivity

If two copies of application logic are maintained for speed and batch layers, then processing capabilities are improved, but development complexity increases

Engineering Contradiction:

Improveprocessing capabilitiesVSAvoiddevelopment complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent merges the application logic into a single unified codebase that operates on Pravega streams. This eliminates the need to maintain separate application logic copies for speed and batch layers, reducing development complexity while preserving full processing capabilities through the universal stream processing interface.

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS11563789B2Executing streaming data writes without duplication or loss

Publication Date: 2023.01.24 EMC IP HLDG CO LLC

AI summary

Implementations are provided herein for executive streaming data writes without duplication or loss. A client application and a pravega node can work to track where write data is, how much data has been written, and what specific data has been acknowledged by the pravega node as successfully written. In the event of an error or connection disruption, the client application can reconnect and determine how much data has been written and resend what data still needs to be written. The data can be written exactly once, and once written and acknowledged, will no longer be subject to data loss.