Parallel Data Stream Processing via Segmented Feed Receivers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face limitations in processing speed due to hardware constraints, data storage methods, and restricted co-processing options, which hinder efficient data processing and query execution.
Innovation Solution
A data stream processing system is implemented, which includes a parallelized data input sub-system, a parallelized data store, retrieve, and/or process sub-system, and a parallelized query and response sub-system. This system processes data in parallel across multiple computing devices, optimizing data storage, retrieval, and query execution through efficient resource allocation and redundancy encoding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional single-threaded data processing is used, then system complexity is low, but processing speed is limited by hardware constraints
Solution Approach 1:
The system segments data processing into multiple independent threads that can execute simultaneously. Each thread handles specific data records or operations, allowing parallel processing that increases speed while distributing complexity across multiple manageable units rather than a single complex monolithic system.
Solution Approach 2:
The patent transitions from single-threaded sequential processing to multi-threaded concurrent processing by adding the time dimension. Multiple threads execute operations at different times simultaneously, effectively utilizing CPU cycles and hardware resources to improve processing speed without proportionally increasing perceived system complexity.
2Speed
If data is stored in traditional formats, then storage simplicity is maintained, but query execution speed is limited
Solution Approach 1:
The data storage structure is segmented into multiple data structures with different access patterns and optimization characteristics. This allows the system to serve different query types efficiently by selecting appropriate segmented structures, improving query execution speed without requiring a single overly complex unified storage system.
Solution Approach 2:
Different portions of the data storage system have different structural qualities optimized for specific operations. Some data structures are optimized for sequential access, others for random access, and others for filtering operations. This local optimization improves overall query execution speed by matching the right storage structure to the right query pattern.
3Productivity
If limited co-processing options are provided, then system simplicity is maintained, but processing efficiency is restricted
Solution Approach 1:
The system implements a universal co-processing framework where threads can perform various operations (reading, writing, filtering, aggregating) on data structures. This multi-functional approach increases processing efficiency by allowing flexible combination of operations while maintaining relative simplicity through a unified co-processing interface rather than requiring separate specialized systems.
Data Source
AI summary
A data stream processing system includes a plurality of feed receiver modules and a central data streaming module. Each of the plurality of feed receiver modules is operable to receive a proper subset of messages of a plurality of incoming messages with keys included in a corresponding subset of a plurality of possible keys assigned to the feed receiver module in key assignment data, and emit only owned ones of the proper subset of messages having keys included in an owned subset of keys in the corresponding subset. The central data streaming module is operable to receive a plurality of messages from the plurality of feed receiver modules that includes only the owned ones of the proper subset of messages emitted via the each feed receiver module, and emit a single data stream that includes the plurality of messages received from the plurality of feed receiver modules.