FPGA Data Processing Device for Large-Scale Query Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current techniques for processing large-scale data in distributed systems like Hadoop struggle with variable-length data and meta information, leading to complex hardware circuits and high costs, which are difficult to support due to size limitations.
Innovation Solution
A data processing device with a processor that generates multiple commands for processing partial data items and transmits them to a hardware circuit, such as an FPGA, to execute processing efficiently, simplifying the hardware circuit and improving processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a hardware circuit is designed to support variable-length data and meta information in distributed systems, then processing capability is improved, but device complexity and cost increase significantly
Solution Approach 1:
The patent divides the data processing task into two segments: the processor handles command generation and high-level control, while the hardware circuit (FPGA) executes specific processing operations. This segmentation allows the hardware circuit to focus on execution without needing to incorporate complex logic for parsing variable-length data formats, meta information structures, and command interpretation, thereby reducing hardware circuit complexity while maintaining processing capability.
Solution Approach 2:
The processor acts as an intermediary between the system and the hardware circuit. It translates high-level processing requests into specific commands that the hardware circuit can execute, and processes the results returned from the hardware circuit. This intermediary role allows the hardware circuit to operate with simpler logic, as it only needs to execute commands and return results without needing to understand the complex data formats and meta information structures.
2Productivity
If many general-purpose data processing servers are used to process large-scale data, then processing capacity is improved, but system size and maintenance cost increase
Solution Approach 1:
The patent combines a processor and a hardware circuit (FPGA) into a single integrated data processing device. This merging allows the system to achieve high processing capacity for large-scale data without needing multiple separate servers, as the hardware circuit provides accelerated processing capabilities within the same physical system, thereby reducing overall system size and maintenance requirements.
Data Source
AI summary
It is possible execute processing large-scale data and improve the processing efficiency while suppressing the complexity of a hardware circuit. A data processing device includes a processor and a FPGA connected to the processor. The processor is configured to acquire a query plan including target identification information identifying data to be processed and a processing detail for the data to be processed, generate, based on the query plan, a plurality of FPGA commands to process a plurality of row group data items constituting the data identified by the target identification information and to be processed, and transmit the FPGA commands to the FPGA. The FPGA is configured to execute processing on the row group data items based on the transmitted FPGA commands and return results of executing the processing to the processor.