Stream-Batch Data Processing Architecture for Early Query Results

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

The lambda architecture in edge computing environments does not assume further speed-up of stream processing, limiting the response time improvement.

Innovation Solution

A data processing system that performs stream processing on received data and batch processing after storage, incorporating a fast response processing layer to divide and process a part of the stream processing, allowing earlier access to processing results.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of time

If stream processing is performed on edge server in lambda architecture, then response time is reduced, but further speed-up is not assumed and processing accuracy may decrease

Engineering Contradiction:

Improveresponse timeVSAvoidprocessing result accuracy

Core Design Contradiction:

Loss of timeVSMeasurement precision

Solution Approach 1:

The patent segments stream processing into two distinct layers: a fast response processing layer that performs simplified processing for quick results, and a real-time processing layer that performs comprehensive processing for accurate results. This segmentation allows the system to provide different levels of processing accuracy based on response time requirements, resolving the contradiction between speed and precision.

Inventive Principle:

Principle #1Segmentation

2Adaptability or versatility

If application is divided into multiple logics and arranged on different machines, then distributed operation is achieved, but large burden is imposed on developer

Engineering Contradiction:

Improvedistributed operation capabilityVSAvoiddevelopment burden

Core Design Contradiction:

Adaptability or versatilityVSEase of manufacture

Solution Approach 1:

The patent introduces a universal stream processing framework that handles distributed operation automatically. The framework provides standardized interfaces and automatic distribution mechanisms, allowing applications to run distributedly without requiring developers to manually divide and arrange logic across machines. This universal framework reduces development burden while maintaining distributed operation capabilities.

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Productivity

If fast response processing is performed on received data, then processing speed is increased, but processing result is less accurate

Engineering Contradiction:

Improveprocessing speedVSAvoidprocessing result accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The patent implements a dynamic processing architecture where the system can switch between fast response mode and real-time processing mode based on requirements. The fast response processing layer uses dynamic simplification of processing logic to achieve high speed, while the real-time processing layer performs complete processing when accuracy is prioritized. This dynamic approach allows the system to adapt processing depth to specific needs.

Inventive Principle:

Principle #15Dynamics

Data Source

PatentUS12450087B2Data processing system that performs stream processing on received data and performs batch processing on the received data after storage, a first processing apparatus, and a non-transitory computer-readable storage medium

Publication Date: 2025.10.21 NT T INC

AI summary

A data processing system includes a first processing apparatus that performs a first phase of stream processing on received data; a first result providing apparatus that stores a first processing result of the first processing apparatus and replies to a query; a second processing apparatus that performs a second phase of the stream processing on the first processing result received from the first processing apparatus; a second result providing apparatus that stores a second processing result of the second processing apparatus and replies to the query; a storage that stores the received data; a third processing apparatus that performs batch processing on the received data read from the storage at a predetermined timing; and a third result providing apparatus that stores a third processing result of the third processing apparatus and replies to the query.