FPGA Hardware Parser for Genomic Data Speed
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The sequential parsing of large genomic data in the SAM format is inefficient due to its variable line lengths and complex branching, leading to performance bottlenecks in software tools, especially on multiprocessor systems with limited hardware resources.
Innovation Solution
A two-stage parsing method is employed, where a hardware parser accelerates the conversion of input data sets by parsing lines up to a threshold size and leaving larger lines unparsed, with error markers, and a software parser completes the conversion of these unparsed lines, utilizing a field programmable gate array (FPGA) or similar hardware for initial parsing and software for error correction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If sequential software parsing is used to handle all data, then parsing accuracy is maintained, but parsing speed deteriorates due to performance bottlenecks
Solution Approach 1:
The parsing process is divided into two segments: a hardware parsing stage that handles the majority of data conversion from SAM format to binary format, and a software parsing stage that processes only the unparsed portions. This segmentation allows the system to leverage hardware acceleration for speed while maintaining software-based accuracy for complex cases.
Solution Approach 2:
An intermediate output format is introduced that contains both parsed and unparsed portions, serving as a bridge between hardware and software parsing stages. This intermediary representation allows the hardware parser to quickly process straightforward cases while flagging complex cases for software handling, thus resolving the speed-accuracy tradeoff.
2Productivity
If hardware acceleration is applied to parse all data, then parsing speed is improved, but hardware resource limitations cause processing failures for large data sets
Solution Approach 1:
The hardware parser performs partial action by processing only those portions of the input data that can be handled within its resource constraints. Lines exceeding the threshold size are left unparsed and flagged for software processing, allowing the system to achieve speed benefits for the majority of data without overwhelming hardware resources.
Solution Approach 2:
The system dynamically adjusts the parsing threshold parameter based on available hardware resources. By changing this parameter, the system can optimize the division of labor between hardware and software parsers, allowing the hardware to handle up to a certain data size threshold while adapting to varying resource availability.
3Device complexity
If a single-stage parsing approach is used, then system complexity is reduced, but overall parsing time increases due to sequential processing bottlenecks
Solution Approach 1:
The parsing system is segmented into distinct hardware and software stages, each optimized for specific types of processing. The hardware stage handles rapid conversion of standard cases, while the software stage handles complex cases, creating a multi-stage system that reduces total parsing time despite increased architectural complexity.
Solution Approach 2:
The hardware parser performs preliminary action by pre-processing and converting the majority of data before software processing begins. This preliminary conversion of straightforward cases eliminates the need for software to process every line sequentially, significantly reducing overall parsing time.
Data Source
AI summary
Methods and systems for accelerated input data conversion include partially parsing an input data set to convert the data set from a first format to a second format in an intermediate output having at least one unparsed portion to quickly perform a majority of the conversion. The partial parsing operates on portions of the input data set having a size less than a threshold size and leaves portions of the input data having a size greater than the threshold size unparsed. The intermediate output is parsed to convert at least one unparsed portion from the first format to the second format in a final output to complete the conversion such that a combined parsing time of the partial parse of the input data set and the parse of the intermediate output is accelerated relative to a single-stage parsing.