Compressed Data Partitioning for Independent Decompression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for handling large data objects in distributed storage systems are inefficient due to the division of data into equal portions, which ignores structural features and introduces dependencies, leading to increased network traffic and loss of association with the original data object.
Innovation Solution
A technique that splits data objects into portions at boundaries within the object, transforming them into segments for independent processing on multiple computing nodes, and stores decompression states to enable independent decompression of compressed data portions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If compressed data is divided into equal portions for distributed storage, then storage efficiency is improved, but the ability to independently decompress portions is lost
Solution Approach 1:
The patent divides compressed data into segments at boundaries that preserve decompression independence. Each segment contains complete decompression state information (dictionary, position, length) needed to decompress it autonomously, eliminating the need to retrieve entire compressed files while maintaining storage efficiency through distributed segment storage.
Solution Approach 2:
The patent performs preliminary decomposition of compressed data into independently decompressible segments before distribution. By pre-calculating and embedding decompression state information (dictionary, position, length) into each segment during the segmentation process, the system enables later independent decompression without requiring access to other segments or the original compressed file.
2Ease of manufacture
If data objects are divided into equal portions, then distribution simplicity is improved, but structural features are ignored leading to increased network traffic
Solution Approach 1:
The patent applies local quality by making each data segment self-contained with its own decompression state information. This allows different segments to be processed independently based on their local characteristics, enabling selective retrieval of only necessary segments for specific queries rather than transferring entire data objects, thus reducing network traffic while maintaining distribution simplicity.
Solution Approach 2:
The patent segments data objects into independently processable units with complete decompression capabilities. This segmentation allows the system to distribute simple portions while enabling selective access to only the segments needed for any given query, avoiding the network traffic overhead of transferring complete data objects or relying on complex inter-segment dependencies.
3Quantity of substance
If compressed data portions are stored without decompression state, then storage space is improved, but decompression independence is lost requiring access to previous portions
Solution Approach 1:
The patent uses copying by duplicating essential decompression state information (dictionary, position, length) into each segment. This copying of critical decompression context allows each segment to be independently decompressed without requiring access to previous portions or the original compressed file, achieving decompression independence while maintaining efficient storage through selective segment distribution.
Data Source
AI summary
A technique of partitioning compressed data includes splitting the compressed data into multiple portions. The technique further includes storing a decompression state in association with a current portion, wherein the decompression state is based on data of a previous portion and enables decompression of the current portion independently of other portions.