Parallel Longest Common Subsequence Engine for Superscale Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional longest common subsequence (LCS) algorithms, such as Hirschberg's algorithm, are impractical for superscale data sequences due to memory constraints and slow processing times, especially when dealing with large datasets like 10 billion event sequences in cloud data centers or 3 billion DNA sequences, as they are designed for single-machine execution and have quadratic time complexity.
Innovation Solution
The approach involves dividing superscale data sequences into segments, allowing for parallel processing using an enhanced longest common subsequence (ELCS) engine, which receives and divides sequences, processes segment pairs using algorithms like Hirschberg's, and merges results to determine the longest common subsequence efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional LCS algorithms like Hirschberg's algorithm are used, then the algorithm can solve the LCS problem in quadratic time and linear space on a single machine, but the program cannot load all data events and sequences into memory for superscale data sequences
Solution Approach 1:
The patent divides superscale data sequences into multiple segments that can be processed in parallel across multiple machines. Each segment is independently processed by separate LCS algorithm instances, allowing the system to handle data sequences larger than available memory by distributing the loading and processing across multiple computational nodes.
Solution Approach 2:
The patent transitions from single-machine sequential processing to multi-machine parallel processing, adding a spatial dimension to the computation. By distributing segments across multiple machines in a cluster, the system overcomes the memory limitations of individual machines while maintaining the ability to compute LCS for superscale data sequences.
2Measurement precision
If Hirschberg's algorithm is used for superscale data sequences, then the algorithm provides optimal sequence alignment, but the quadratic time complexity makes it slow and time consuming
Solution Approach 1:
The patent segments the superscale data sequences into smaller subsequences that can be processed in parallel. By dividing the overall LCS problem into multiple independent segment pairs, the system reduces the time complexity from O(n²) for the entire sequence to O((n/k)² × k) = O(n²/k) where k is the number of parallel segment pairs, thereby reducing total processing time while maintaining alignment accuracy through the merging of segment results.
Solution Approach 2:
The patent performs preliminary segmentation and filtering of the data sequences before applying the LCS algorithm. By pre-processing the superscale sequences to identify and segment relevant portions, the system reduces the effective input size for the quadratic-time LCS algorithm, thereby reducing overall processing time while preserving the accuracy of the alignment for the segments that are processed.
3Reliability
If conventional LCS algorithms are applied to superscale data sequences, then the algorithm provides a solution, but there is too much noise for practical use if the sequences are not naturally well aligned
Solution Approach 1:
The patent extracts and removes noisy or irrelevant portions of the superscale data sequences before processing. By filtering out segments that are unlikely to contribute meaningfully to the LCS or that contain excessive noise, the system reduces the amount of harmful information that would otherwise pollute the results, thereby improving the validity and practical usefulness of the computed LCS for superscale sequences.
Data Source
AI summary
Enhanced longest common subsequence techniques are provided to improve efficiency and accuracy of any process that uses long sequences. Two superscale data sequences are received. Each data sequence is divided into a set of segments based on one or more heuristics or user input. From the sets of segments, pairs of segments are selected and processed using any longest common subsequence algorithm to determine the longest common subsequence for each pair of segments. The results of the processing are merged to determine the enhanced longest subsequence which is the longest common subsequence for the received superscale data sequences. Dividing the superscale data sequences into segments allows for parallel processing of each segment pair, which results in a faster completion time than prior art methods for determining longest common subsequences.