Parquet File Reader Hardware for Parallel Column Reconstruction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in efficiently processing Parquet files due to their complex hierarchical structure and nested data formats, leading to inefficient data readout and high computational overhead.
Innovation Solution
A hardware-implemented file reader system comprising column readers and a record reconstructor, which decompresses and aligns data from multiple columns in parallel, utilizing backpressure mechanisms to maintain alignment and reconstruct records efficiently, while supporting dictionary translation and value manipulation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional software-based Parquet processing is used, then flexibility and ease of operation are maintained, but processing speed and throughput are significantly reduced
Solution Approach 1:
The system divides the Parquet file processing into independent column-based units, with each column reader handling a specific column separately. This segmentation enables parallel processing of multiple columns simultaneously, significantly improving throughput while maintaining manageable complexity through modular design
Solution Approach 2:
The patent replaces traditional software-based processing with a dedicated hardware circuit implementation. The hardware circuit directly processes Parquet files by reading compressed column data, decompressing it, and reconstructing records without relying on software interpreters, thereby achieving much faster processing speeds
2Productivity
If all column data is read and decompressed, then complete record reconstruction is achieved, but unnecessary data processing increases time and resource consumption
Solution Approach 1:
The system extracts only the necessary columns and data required for record reconstruction, rather than processing entire columns. By selectively reading and decompressing only the minimal required data from each column, the system eliminates unnecessary data processing operations, reducing both time and computational resource consumption
Solution Approach 2:
The hardware circuit performs partial decomposition and processing of column data, reading only the necessary portions required for record reconstruction. This partial action approach avoids the excessive processing of complete columns when only subsets are needed, optimizing the balance between processing completeness and efficiency
3Productivity
If column readers operate independently at full speed, then throughput is maximized, but alignment between column readers becomes difficult to maintain
Solution Approach 1:
The system implements feedback mechanisms where column readers receive signals about their progress and the progress of other column readers. This feedback enables dynamic adjustment of reading speeds and synchronization points, allowing independent high-speed operation while maintaining precise alignment through continuous coordination based on real-time status information
Data Source
AI summary
A hardware-implemented file reader includes an interface, multiple hardware-implemented column readers and a hardware-implemented record reconstructor. The interface is configured to access a file including multiple records. The records store values in accordance with a nested structure that supports optional values and repeated values. The file is stored in a columnar format having multiple columns, each column storing (i) compressed values and (ii) corresponding compressed structure information that associates the values in the column to the nested structure of the records. Each column reader is configured to be assigned to a respective selected column, and to read and decompress both the values and the structure information from at least a portion of the selected column. The record reconstructor is configured to reconstruct one or more of the records from at least portions of the columns that are read by the column readers, and to output the reconstructed records.