Distributed Streaming Data Processing for Sub-Second OLAP Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in processing large, constantly changing, low-level data sets in real-time due to inefficiencies in handling sub-second response times for flexible queries on high-volume, streaming data, particularly in wireless telecommunication networks.
Innovation Solution
A distributed system that processes streaming data records by reorganizing them into OLAP cubes across multiple computational nodes, allowing for sub-second query response times by parsing and redistributing data based on dimensions, creating targeted subsets, and updating databases accordingly.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional data warehouse systems are used to store and query data, then data can be organized into structured formats suitable for analysis, but query response time exceeds sub-second requirements
Solution Approach 1:
The system pre-computes and stores aggregated data in OLAP cubes before queries are submitted. Data is organized into dimension-based cubes (e.g., device-type, application, region) with pre-calculated measures, allowing instant retrieval without real-time computation. This preliminary organization enables sub-second query response times while maintaining the ability to handle flexible ad-hoc queries.
2Measurement precision
If OLAP cubes are used to organize data for fast querying, then query response time improves to sub-second levels, but the system cannot efficiently handle low-level high-volume streaming data
Solution Approach 1:
The system segments the data processing into two distinct stages: (1) a stream processing stage that ingests and buffers low-level high-volume streaming data in its raw format, and (2) a cube construction stage that periodically aggregates this data into OLAP cubes. This segmentation allows the system to maintain flexibility in handling diverse streaming data formats while producing structured OLAP cubes for fast querying.
Solution Approach 2:
The patent introduces an intermediary buffer layer between the streaming data source and the OLAP cube structure. This buffer stores raw streaming records temporarily and serves as a transition zone where data is gradually aggregated into cube structures. The intermediary allows the system to accept flexible streaming inputs while producing structured outputs suitable for fast OLAP queries.
3Adaptability or versatility
If data is stored in disaggregated low-level format, then data flexibility and query adaptability are maintained, but processing speed and response time deteriorate
Solution Approach 1:
The system employs periodic batch processing to transform disaggregated streaming data into aggregated OLAP cube structures at regular intervals. Between cube construction cycles, the system accepts and buffers streaming data in flexible formats. When cubes are constructed periodically, the aggregated data enables fast querying. This periodic transformation maintains query flexibility during data ingestion while achieving fast response times during query execution on pre-aggregated data.
Data Source
AI summary
Representative embodiments of a distributed processing method of facilitating interactive analytics of streaming data records by receiving the data records at a plurality of distributed computational nodes, establishing and storing dimensions corresponding to attributes of the data records, parsing the streaming data records to identify matches to at least one of the dimensions and based thereon, reducing the number of data records to create a targeted subset of the data, re-distributing the targeted subsets of the streaming data records among the distributed computational nodes in accordance with the dimensions stored on the nodes, updating a database storing measures of the dimensions in accordance with the targeted subsets of the streaming data records, and using the database to respond to a query based on measures associated with one or more of the dimensions.