Tokenized Stream Compression Using Column-Wise Radix Sorting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression techniques, such as the bzip2 compressor using the Burrows-Wheeler Transform, are inefficient for tokenized streams, leading to high infrastructure costs for log storage and analysis due to suboptimal sorting algorithms that require excessive memory and cause cache misses.
Innovation Solution
A compression sorter that rearranges tokenized streams by applying sort orders column-wise, based on previous columns' orders, using radix sorting to improve efficiency and reduce memory requirements, allowing for faster sorting and practical application with larger block sizes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If the Burrows-Wheeler Transform sorting algorithm is used in the bzip2 compressor, then data compression can be achieved, but memory usage increases and cache misses occur
Solution Approach 1:
The patent divides the tokenized stream into multiple columns (c columns) where each column represents a specific position in the tokens. This segmentation allows the sorting algorithm to process smaller, manageable portions of data independently, reducing the memory footprint compared to traditional sorting methods that require loading entire datasets into memory.
Solution Approach 2:
The patent transforms the one-dimensional sorting problem into a multi-dimensional columnar structure. By organizing tokens into c columns and arranging them in a grid-like fashion, the algorithm can perform sorting operations along different dimensions, improving cache utilization and reducing memory access patterns that cause cache misses.
2Loss of information
If the Burrows-Wheeler Transform sorting algorithm is used, then data can be sorted for compression, but processing speed decreases due to excessive memory access
Solution Approach 1:
By segmenting the sorting process into column-wise operations, the patent enables parallel processing of multiple columns and reduces the working set size for each sorting operation. This segmentation minimizes memory access overhead and improves processing speed.
Solution Approach 2:
The patent changes the sorting parameters by using column-based sorting with c columns instead of traditional row-based sorting. This parameter change allows the algorithm to optimize for cache-friendly access patterns and reduce the number of memory accesses required during sorting operations.
3Loss of information
If traditional sorting algorithms are used on tokenized streams, then sorting can be performed, but infrastructure costs increase due to suboptimal performance
Solution Approach 1:
The patent optimizes sorting performance by changing fundamental parameters of the sorting approach - using columnar organization with c columns and position-based sorting. These parameter changes lead to better compression ratios and faster processing, directly reducing infrastructure costs for log storage and analysis.
Solution Approach 2:
The patent creates a compressed representation of the tokenized stream through columnar sorting, producing a more compact output that requires less storage space. This copying/transformation approach achieves better compression efficiency, reducing the overall storage infrastructure requirements.
Data Source
AI summary
A tokenized stream including n tokens, each token including two or more portions, is received and a first sort order based on a sort of a set of the first portions of the n tokens is determined. The first sort order is applied to reorder a set of the second portions of the n tokens. The above steps are repeated to determine a sort order based on a set of portions of the n tokens and to apply the sort order to another set of portions of the n tokens column until a cth set of portions been reordered by a (c−1)th sort order. The variable c is a desired number of sets to be sorted. The variables c and n are whole numbers and the n tokens are dispersed during reordering.