Text String Dictionary Blocking for Multicore Parallel Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for generating a text string dictionary, especially for large DNA sequence data, are limited by the parallelization degree, which restricts the effective use of multiple CPU cores, leading to reduced processing speed due to unbalanced CPU core loads and synchronization issues.
Innovation Solution
The method involves dividing the text string dictionary into blocks with unique labels, including delimiters, and executing entry registration processes in parallel across these blocks, allowing for efficient distribution of text string data and synchronization, thereby increasing the parallelization degree and processing speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If parallelization is performed for each base type (A, C, G, T), then processing speed increases approximately 4 times, but parallelization degree is limited to alphabet size
Solution Approach 1:
The text string dictionary is divided into multiple blocks, each block containing a subset of the sorted suffixes. This segmentation allows parallel processing across blocks while maintaining the ability to handle large datasets, thereby increasing both processing speed and parallelization degree beyond the alphabet size limitation.
2Productivity
If number of partial processes equals number of CPU cores, then processor utilization is maximized, but calculation load balance becomes difficult
Solution Approach 1:
The dictionary is divided into multiple blocks that can be processed in parallel. Each block is further divided into subsets that can be assigned to different CPU cores, enabling flexible load distribution that balances computation across cores while maintaining high processor utilization.
Solution Approach 2:
The parallel processing architecture dynamically assigns and redistributes computational tasks across available CPU cores based on current load conditions. This dynamic approach allows the system to adapt to varying computational demands and maintain balanced loads across cores.
3Adaptability or versatility
If more blocks are created for parallel processing, then parallelization degree increases, but synchronization wait time increases
Solution Approach 1:
The dictionary is divided into blocks that can be processed independently in parallel. By carefully managing the granularity of segmentation and using efficient synchronization mechanisms, the system achieves high parallelization while minimizing the overhead and wait times associated with coordinating multiple parallel processes.
Data Source
AI summary
A multicore CPU of a text string data analyzing device: loads a plurality of blocks obtained by dividing a text string dictionary into a memory; executes, in parallel on block groups executable independently of each other, an entry registration process of registering, character by character, unregistered text strings of text string data as new entries in the blocks in order from last characters; and outputs, as BW transformed data of the text string dictionary in which the text string data is already registered, a text string obtained by coupling text strings registered in entries of the blocks in a state in which no unregistered text strings of the blocks exists.