Data Format Conversion Circuits With Fine-Grained OLAP Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems face inefficiencies in processing large datasets due to the need for transcoding between storage optimized and processing optimized data formats, particularly in online analytical processing (OLAP) workloads, where only a subset of data is typically required for analysis, leading to suboptimal performance and increased costs.
Innovation Solution
A hardware engine is introduced that transcodes data from storage optimized formats like Apache Parquet to processing optimized formats like Apache Arrow, incorporating fine-grained filtering capabilities to efficiently extract relevant data for processing, thereby improving performance and reducing costs by offloading data transcoding and filtering tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in storage optimized formats (e.g., Apache Parquet), then storage efficiency is improved, but processing speed and performance deteriorate due to the need for transcoding
Solution Approach 1:
The patent applies preliminary action by pre-filtering data during the storage phase. Filter expressions are evaluated and stored alongside the data in columnar format, so that when data is retrieved for processing, the filtering has already been performed, eliminating the need for post-retrieval filtering and reducing processing time
Solution Approach 2:
The patent introduces an intermediary filtering mechanism that operates between storage and processing. The filter circuitry acts as an intermediary layer that can evaluate filter expressions against stored data without requiring full transcoding, thereby maintaining storage efficiency while improving processing speed
2Speed
If all data is transcoded to processing optimized formats (e.g., Apache Arrow), then processing performance is improved, but resource consumption and costs increase
Solution Approach 1:
The patent applies the extraction principle by selectively extracting and transcoding only the subset of data that matches filter criteria, rather than transcoding entire datasets. The filter circuitry identifies and extracts only relevant rows from columnar storage, converting them to processing optimized format on-demand, thereby reducing resource consumption while maintaining processing performance for relevant data
Solution Approach 2:
The patent applies local quality by applying different processing treatments to different portions of data. Filtered data that matches criteria is transcoded to processing optimized format, while data that doesn't match remains in storage optimized format, creating a heterogeneous processing approach that optimizes resource usage based on local data characteristics
3Productivity
If fine-grained filtering is implemented during transcoding, then data extraction efficiency is improved, but device complexity increases
Solution Approach 1:
The patent applies merging by combining the transcoding function and filtering function into a single integrated circuit device. The transcoder circuitry and filter circuitry work together in a unified architecture, allowing fine-grained filtering during transcoding without requiring separate processing stages, thereby improving data extraction efficiency while limiting the increase in device complexity through functional integration
Data Source
AI summary
An integrated circuit includes conversion circuitry for converting first data in a first data format optimized for efficient data storage into second data in a second data format optimized for processing by a processing circuit. The integrated circuit also includes filter circuitry for filtering the second data to generate filtered data in the second data format. The integrated circuit outputs the filtered data for processing by the processing circuit.