Unified Stream Batch Processing via Partition Files

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current stream processing systems require high resource costs for real-time data processing, while batch processing systems offer lower resource costs but with high latency, necessitating separate dedicated resources for both, which is costly and inefficient.

Innovation Solution

Implementing a near-real-time stream processing system that uses the same distributed file system as batch processing, where data is processed in partition files with defined windows and lifetime windows, allowing for near-real-time processing without duplicating data and reducing resource needs.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If a stream processing system is implemented with dedicated messaging queues and storage, then real-time data processing capability is improved, but resource cost and engineering cost increase significantly

Engineering Contradiction:

Improvedata processing speedVSAvoidresource cost

Core Design Contradiction:

SpeedVSQuantity of substance

Solution Approach 1:

The patent merges stream processing and batch processing into a single unified system that uses the same distributed file system for both workloads. The system combines real-time processing capabilities with batch processing infrastructure, eliminating the need for separate dedicated resources for stream processing while maintaining near-real-time performance through partition-based processing and time-window management.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The distributed file system is designed to serve multiple functions simultaneously - it acts as both the storage backend for batch processing and the event source for stream processing. The same file system infrastructure supports both batch jobs that process historical data and stream processing jobs that handle real-time data flows, making the system multi-functional and reducing overall resource requirements.

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Quantity of substance

If a batch processing system is used, then resource cost is reduced, but data processing latency increases to hours or days

Engineering Contradiction:

Improveresource costVSAvoiddata processing latency

Core Design Contradiction:

Quantity of substanceVSLoss of time

Solution Approach 1:

The patent segments the data processing workload into different time windows and partition files. By organizing data into time-based partitions and processing them in manageable segments, the system can achieve near-real-time processing for recent data while using batch processing for historical data. This segmentation allows the system to reduce latency for time-sensitive operations without requiring full stream processing resources.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system dynamically adjusts processing behavior based on data recency and processing requirements. For recent partition files, the system applies stream processing techniques with lower latency, while for older data, it uses traditional batch processing. This dynamic approach allows the system to optimize latency for critical operations while maintaining cost efficiency for less time-sensitive workloads.

Inventive Principle:

Principle #15Dynamics

3Adaptability or versatility

If separate stream processing and batch processing systems are maintained, then both real-time and comprehensive insights are achieved, but engineering cost and resource cost double

Engineering Contradiction:

Improveprocessing capabilityVSAvoidresource cost

Core Design Contradiction:

Adaptability or versatilityVSQuantity of substance

Solution Approach 1:

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

Inventive Principle:

Principle #6Universality (Multi-functionality)

4Loss of time

If data is processed event-by-event in real-time, then processing latency is reduced, but processing resources and storage resources increase significantly

Engineering Contradiction:

Improveprocessing latencyVSAvoidprocessing resources

Core Design Contradiction:

Loss of timeVSQuantity of substance

Solution Approach 1:

The patent applies partial real-time processing by processing data in time-based partitions rather than strictly event-by-event. The system processes recent partitions with lower latency using stream processing techniques, while allowing batch processing for older partitions. This partial application of real-time processing reduces the need for continuous high-resource allocation while maintaining acceptable latency for time-sensitive operations.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentEP4292269B1Near-real-time data processing with partition files

Publication Date: 2024.10.09 MICROSOFT TECHNOLOGY LICENSING LLC

Figure 1
Figure 2A
Figure 2B

AI summary

Embodiments disclosed herein are related to implementing a near-real-time stream processing system using the same distributed file system as a batch processing system. A data container and partition files are generated according to a partition window that specifies a time range that controls when data is to be included in the partition files. The data container is scanned to determine if the partition files are within a partition lifetime window that specifies a time range that controls how long the partition files are active for processing. For each partition file within the lifetime window, processing tasks are created based on an amount of data included in the partition files. The data in the partition files is accessed and the processing tasks are performed. Information about the partition files is recorded in a configuration data store.