Unified DSP Query Processor for Streaming Data Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing large amounts of data from sensor networks across different architectures is computationally expensive and inefficient, requiring multiple technologies for online and offline data analysis, which leads to high latency and unnecessary data transfer.
Innovation Solution
Deep integration of digital signal processing (DSP) operations with a general-purpose query processor, providing a unified query language for tempo-relational and signal data, and supporting incremental computation to reduce data transfer and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple technologies are used for online and offline data analysis, then different data architectures can be supported, but computational cost and processing time increase
Solution Approach 1:
The patent merges digital signal processing operations with query processing by integrating DSP operators into the query execution framework. This allows unified processing of temporal-relational and signal data using a single technology stack, eliminating the need for multiple separate technologies and reducing computational overhead while maintaining support for both online and offline data architectures.
Solution Approach 2:
The query processor is designed to universally handle both temporal-relational queries and signal processing operations through a unified language and execution engine. The same query processing infrastructure can execute relational operations, signal processing operations, and hybrid operations, providing multi-functionality that reduces the need for specialized separate systems.
2Device complexity
If data is transferred between separate query databases and DSP processors, then processing can be distributed, but data transfer overhead increases
Solution Approach 1:
The patent combines query processing and DSP operations within a single integrated processor, eliminating the need for data transfer between separate systems. The unified architecture allows query operators to directly execute DSP operations on data without intermediate transfer steps, reducing communication overhead while maintaining distributed processing capabilities through the query execution framework.
3Reliability
If conventional separate technologies are used for query processing and DSP, then each technology can be optimized for its specific task, but overall processing latency increases
Solution Approach 1:
The patent merges the execution engines of query processing and DSP operations into a unified system. This integration allows for tighter coupling of operations, reducing the latency between query formulation and result delivery by eliminating intermediate processing steps and data transfer delays that would occur with separate technologies.
Solution Approach 2:
The unified query processor can perform preliminary processing and optimization of data and operations before execution, including pre-computation of DSP operators and optimization of query plans. This preliminary action reduces the overall processing time by preparing computational resources and data structures in advance, while maintaining the optimization benefits of having both query and DSP capabilities in one system.
Data Source
Figure 1
Figure 2
Figure 3A~3B
AI summary
The techniques and systems described herein are directed to providing deep integration of digital signal processing (DSP) operations with a general-purpose query processor. The techniques and systems provide a unified query language for processing tempo-relational and signal data, provide mechanisms for defining DSP operators, and support incremental computation in both offline and online analysis. The techniques and systems include receiving streaming data, aggregating and performing uniformity processing to generate a uniform signal, and storing the uniform signal in a batched columnar representation. Data can be copied from the batched columnar representation to a circular buffer, where DSP operations are applied to the data. Incremental processing can avoid redundant processing. Improvements to the functioning of a computer are provided by reducing an amount of data that to be passed back and forth between separate query databases and DSP processors, and by reducing a latency of processing and/or memory usage.