Actor-Based Stream Processing with Batched Table Ingestion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large networks with numerous physical devices face challenges in efficiently processing and storing data due to latency and low throughput, leading to incomplete data analysis, especially when data needs to be processed within a specific time frame.
Innovation Solution
Implementing an actor-based data processing system that uses stream partitions, a stream partition manager, and a merge manager to distribute and process data packets in batches, reducing contention and improving throughput by using bulk loading techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If data is transmitted directly from physical devices to data collection service, then data transmission is simple, but latency increases and throughput decreases
Solution Approach 1:
The patent introduces a data collector as an intermediary component between physical devices and the persistent store. This mediator aggregates data from multiple devices and performs batched writes to the persistent store, thereby reducing the number of individual transmission requests and improving throughput while maintaining simple data collection at the device level.
Solution Approach 2:
The patent segments the data processing workflow into distinct phases: data collection from devices, data aggregation at the collector, and batched persistence operations. This segmentation allows the system to handle data in manageable chunks rather than processing every individual data point immediately, reducing latency and improving overall throughput.
2Device complexity
If data is written directly to persistent store from each physical device, then data storage is straightforward, but processing efficiency decreases and throughput is low
Solution Approach 1:
The patent merges multiple individual data write operations into a single batched write operation at the data collector. Instead of each physical device writing its data independently to the persistent store, the collector aggregates data from all devices and performs a consolidated write, thereby reducing the total number of write operations and improving processing throughput.
Solution Approach 2:
The data collector performs preliminary aggregation and buffering of data before writing to the persistent store. It collects data from multiple devices in advance and accumulates it in memory, then performs batched writes to the persistent store. This preliminary action reduces the frequency of write operations and improves overall processing efficiency.
3Productivity
If batched data processing is implemented, then throughput improves, but data processing time increases
Solution Approach 1:
The patent implements dynamic batching where the data collector adjusts the batch size and write frequency based on system conditions. It can process data in smaller batches more frequently or aggregate larger batches less frequently, depending on the requirements. This dynamic approach allows the system to optimize between throughput and latency by adapting to changing workloads and priorities.
Solution Approach 2:
The data collector performs periodic batched writes to the persistent store rather than continuous individual writes. It aggregates data over a time period and then performs a batched write operation. This periodic action smooths out the write operations, improving throughput by reducing the number of individual write calls while maintaining acceptable latency through timely batched processing.
Data Source
AI summary
A method includes receiving data packets from a stream partition of a plurality of stream partitions, processing the data packets with a plurality of stream processors to generate multiple table entries, transmitting the multiple table entries in batches to a plurality of staging tables, the plurality of staging tables being provided for a respective one of the plurality of stream partitions, and receiving the batches at a target table communicatively coupled to the plurality of staging tables, the batches being transmitted from the plurality of staging tables. The plurality of staging tables are updated in first batches at a first frequency and the target table is updated in second batches at a second frequency, the second frequency being different than the first frequency.