High-Throughput Sequencing Data Analysis Using Distributed Suffix-Expansion Sorting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for high-throughput sequencing data analysis are limited by system memory and disk size, or require pre-partitioning data into smaller subsets, which restricts the ability to obtain global insights across all data points, failing to efficiently process and analyze large volumes of genomic sequencing data.
Innovation Solution
A method and system for high-throughput sequencing data analysis that involves partitioning input DNA/RNA/Protein sequences into overlapping segments, distributing them across multiple slave computing units in a cloud environment, and performing suffix-expansion-sorting processing to construct a global database table, enabling efficient analysis and scalability as data grows.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is pre-partitioned into smaller subsets for processing, then processing speed is improved, but the ability to obtain global insights across all data points deteriorates
Solution Approach 1:
The patent divides the large sequencing data into multiple partitions that are distributed across different computing nodes. Each node processes its local partition independently, achieving parallel processing speedup. The key innovation is that despite segmentation, the system maintains the ability to perform global operations by coordinating across partitions through a unified query interface and distributed data structures.
Solution Approach 2:
The patent merges results from multiple distributed partitions to provide global insights. The distributed database system combines local processing results through coordinated queries, allowing users to obtain comprehensive analytics across all data points while maintaining the performance benefits of distributed processing.
2Device complexity
If single processing unit is used, then system complexity is reduced, but processing capacity is limited by memory and disk size
Solution Approach 1:
The patent transitions from a single processing unit to a distributed multi-dimensional architecture. Instead of expanding memory and disk on one machine, the system distributes data and computation across multiple nodes in a networked architecture, effectively adding spatial dimensionality to the processing capacity.
Solution Approach 2:
The distributed database system provides a unified interface that abstracts the underlying complexity of multiple processing units. Users interact with a single logical database system that automatically distributes queries across multiple physical nodes, making the system appear simple while delivering massive processing capacity.
3Loss of information
If data volume increases, then analytical value is improved, but processing efficiency deteriorates
Solution Approach 1:
The patent performs preliminary processing and indexing of sequencing data when it is first ingested into the distributed system. Data is pre-processed, validated, and organized into optimized storage formats across the distributed nodes. This preliminary action enables efficient querying and analysis even as data volume grows to petabyte scales.
Solution Approach 2:
The distributed database system dynamically adapts to increasing data volumes by automatically distributing new data across available nodes, balancing workloads, and optimizing query execution plans. The system scales elastically, maintaining processing efficiency as data grows by dynamically adjusting resource allocation and data placement.
Data Source
AI summary
Methods and systems for high-throughput sequencing data analysis are provided. In an embodiment, the method includes the following steps. An input DNA/RNA/Protein sequence is received by a master computing unit. The input DNA/RNA/Protein sequence is partitioned into overlapping segments with a sliding window less than a segment length of the overlapping segments to allow overlapping of any successive two thereof by the master computing unit. The overlapping segments are distributed by the master computing unit to a plurality of slave computing units in a cloud computing environment. Suffix-expansion-sorting processing is performed on the overlapping segments by the slave computing units to produce sorted expansion segments. Distributed database tables are generated based on the sorted expansion segments by at least a portion of the slave computing units. The distributed database tables are associated to construct a global database table corresponding to the input DNA/RNA/Protein sequence for high-throughput sequencing data analysis.