Compressed Data DAG Traversal for Faster Big Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Big data processing faces challenges with large storage overhead and prolonged processing times due to the need for decompression of compressed data, which hinders efficient data analysis and storage.
Innovation Solution
A big data processing method using a modified Sequitur compression algorithm that transforms data into a directed acyclic graph (DAG) for direct processing, allowing for top-downward or bottom-upward traversal to optimize data analysis, particularly suitable for multi-core CPU and GPU environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of stationary object
If data compression algorithms (LZ77, suffix arrays) are used to reduce storage overhead, then storage space is reduced, but decompression steps prolong end-to-end processing time
Solution Approach 1:
The patent applies preliminary compression action using the Sequitur algorithm to transform data into a DAG structure before processing. This pre-compression step creates a hierarchical representation that can be directly processed without requiring traditional decompression, thus reducing storage space while avoiding the time penalty of decompression during data analysis operations.
Solution Approach 2:
The patent replaces the traditional mechanical decompression process with a direct traversal mechanism on the compressed DAG structure. Instead of mechanically decompressing data to access it, the system substitutes this with logical traversal operations (top-downward or bottom-upward) on the pre-compressed hierarchical structure, eliminating the decompression step entirely.
2Quantity of substance
If traditional compression algorithms are used, then storage overhead is reduced, but the decompression steps increase end-to-end processing time
Solution Approach 1:
The patent introduces dynamic traversal patterns that can adapt to different data access patterns and query types. The system can switch between top-downward traversal for certain operations and bottom-upward traversal for others, optimizing processing speed while maintaining compressed data representation. This dynamic approach allows the system to maintain high productivity despite working with compressed data structures.
Solution Approach 2:
The patent changes the fundamental parameter of data representation from traditional linear compressed formats to a hierarchical DAG structure. This parameter change enables direct processing operations on compressed data by transforming the data organization paradigm, allowing traversal-based access patterns that maintain both compression efficiency and processing speed.
3Volume of stationary object
If data is compressed to reduce storage requirements, then storage efficiency improves, but data analysis becomes more complex and time-consuming
Solution Approach 1:
The patent segments data into hierarchical components within the DAG structure, organizing compressed data into manageable nodes and edges. This segmentation allows complex data analysis operations to be broken down into simpler traversal steps, reducing processing complexity while maintaining storage efficiency. The hierarchical structure naturally divides data into segments that can be processed independently and combined.
Solution Approach 2:
The patent introduces the DAG structure as an intermediary representation between raw data and processed results. This intermediary hierarchical structure simplifies complex data analysis operations by providing a structured pathway for traversal and computation, reducing the complexity of working with compressed data while maintaining storage efficiency.
Data Source
AI summary
A big data processing method based on direct computation of compressed data. The method includes 1) compressing, based on a modified Sequitur compression method, original input data according to a smallest compression granularity given by an user, and transforming them into a directed acyclic graph, DAG, consisting of digits; and 2) determining an optimal traversal pattern, and performing a top-downward traversal or a bottom-upward traversal on the DAG in the step 1) based on the determined optimal traversal pattern so as to enable direct processing of the compressed data. By providing a modified Sequitur algorithm and top-downward and bottom-upward traversal strategies in the disclosure, direct processing of compressed data is enabled, significant improvement in time and space has been gained with broad applicability, and certain representations with respect to more advanced document analytics can still be derived on the basis of these.