Parallel Data Stream Processing via Nested Structures
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in efficiently processing large volumes of time-stamped data streams, particularly in analyzing user activity and system data, due to the need for large processing resources and the high cost of parallel database products.
Innovation Solution
A method and system for processing a stream of data in parallel across multiple nodes, utilizing a log module to organize data into sequential and nested structures, and a scalable query processing module that applies Mobius Query Language (MQL) operators to enable automatic parallel execution of relational and sequence queries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If parallel database products are used to process large data streams, then processing capability is improved, but cost increases prohibitively
Solution Approach 1:
The patent segments the parallel database system into modular components including compute nodes, storage nodes, and coordination services. Each node processes data independently, enabling scalable parallel processing without requiring expensive proprietary parallel database systems. The data stream is partitioned across multiple nodes, with each node handling a subset of the processing workload.
Solution Approach 2:
The patent employs commodity hardware nodes instead of expensive parallel database appliances. These nodes can be provisioned, scaled, and decommissioned economically based on processing needs. The system uses standard off-the-shelf components rather than specialized expensive hardware, making parallel processing accessible and cost-effective.
2Adaptability or versatility
If custom code is used to perform data analysis tasks, then flexibility is improved, but complexity increases
Solution Approach 1:
The patent implements a universal query processing framework that handles multiple data analysis tasks through a common interface. The system supports various query types (aggregations, joins, filters, window functions) and data operations through standardized SQL-like syntax, eliminating the need for multiple custom code implementations for different analysis scenarios.
Solution Approach 2:
The patent introduces an intermediary query processing layer that sits between the user and the distributed data storage. This layer translates high-level queries into distributed execution plans, managing the complexity of parallel processing, data partitioning, and node coordination automatically. Users interact with simple query syntax while the intermediary handles the complex distributed execution details.
3Productivity
If data is partitioned into logical sub-streams for analysis, then processing efficiency is improved, but system complexity increases
Solution Approach 1:
The patent automatically partitions data streams into logical sub-streams distributed across multiple nodes based on data characteristics and query patterns. This segmentation enables parallel processing of different data portions simultaneously, improving efficiency without requiring manual intervention. The system manages partitioning, routing, and reassembly automatically.
Solution Approach 2:
The patent implements self-service data partitioning where the system automatically determines optimal partitioning strategies based on workload analysis. The query optimizer automatically selects partitioning keys and distribution strategies without user input, and the system dynamically adjusts partitioning based on observed query patterns and data characteristics.
Data Source
AI summary
A method and a system process a stream of data in parallel across a plurality of nodes. The log processing system has a log module, a query language module, and a query processing module. The log module receives and organizes the stream of data into a sequential and nested data structure. The query language operator module defines operators that operate on the sequential and nested data structure. The query processing module processes in parallel across a plurality of nodes a query based on an operator on the stream of data.