Incremental Burrows-Wheeler Transform for Large Dataset Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data compression methods for large datasets, such as those generated by nucleic acid sequencing, face challenges in efficiently compressing and decompressing data, requiring significant computational resources and memory, which limits their analysis on typical desktop or laptop computers.
Innovation Solution
The Burrows-Wheeler transform (BWT) is computed in a character-by-character cumulative manner to enhance data compression, allowing for incremental building and indexing, which reduces computational demands and memory usage, enabling efficient compression and decompression of large datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If traditional data compression methods are used on large datasets, then compression is achieved, but computational resources and memory usage become excessive
Solution Approach 1:
The patent applies segmentation by dividing the large dataset into smaller blocks or chunks that can be processed independently. The Burrows-Wheeler transform is applied to each block separately, and results are combined. This reduces the memory footprint and computational load at any given time while still achieving effective compression across the entire dataset.
Solution Approach 2:
The patent performs preliminary sorting and reordering of data using the Burrows-Wheeler transform before applying final compression. This preliminary action rearranges data to create patterns that are easier to compress, reducing the computational work needed in subsequent compression steps while maintaining effective compression ratios.
2Loss of substance
If traditional data compression methods are used on large datasets, then compression is achieved, but memory usage becomes excessive
Solution Approach 1:
The patent segments the dataset into manageable blocks that fit within available memory constraints. Each block is processed independently through the Burrows-Wheeler transform and compression pipeline, with results stored incrementally. This approach enables compression of terabyte-scale datasets on systems with limited RAM.
Solution Approach 2:
The patent transitions from processing entire datasets in memory to a streaming/block-based approach that processes data in sequential chunks. This dimensional change in processing architecture allows compression of datasets much larger than available memory by utilizing disk storage and sequential I/O operations.
3Loss of substance
If Burrows-Wheeler transform is applied to large datasets, then compression ratio improves, but computational time increases
Solution Approach 1:
The patent divides the large dataset into multiple smaller blocks and applies the Burrows-Wheeler transform to each block in parallel or sequential order. This segmentation reduces the computational complexity from O(n²) for the entire dataset to sum of O(ni²) for individual blocks, where ni is the size of each block, significantly reducing total computational time.
Solution Approach 2:
The patent applies the Burrows-Wheeler transform to only the necessary portions of data at each processing stage rather than reprocessing entire datasets. Intermediate results are preserved and built upon, avoiding redundant computational work while still achieving the compression benefits of the transform.
Data Source
AI summary
The present disclosure provides computer implemented methods and systems for analyzing datasets, such as large data sets output from nucleic acid sequencing technologies. In particular, the present disclosure provides for data analysis comprising computing the BWT of a collection of strings in an incremental, character by character, manner. The present disclosure also provides compression boosting strategies resulting in a BWT of a reordered collection of data that is more compressible by second stage compression methods compared to non-reordered computational analysis.