Compressed Data Chunk Mapping for Fast Ranged Reads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional techniques for accessing compressed data objects require decompressing all preceding data to access a specific portion, leading to computational inefficiency and high time consumption, as they do not enable decompression to start at a specific requested position without determining the compression of preceding data.
Innovation Solution
A computer system compresses data into chunks and uses a map data structure to associate compressed and uncompressed positions, allowing decompression to initiate at the requested offset within the compressed data object, optimizing ranged reads by compacting the map to fit entirely in memory and minimizing its size.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If all preceding data is decompressed to access a specific portion of compressed data, then the desired data portion can be accessed, but computational efficiency and time consumption deteriorate
Solution Approach 1:
The patent divides the compressed data object into multiple chunks, each with its own compression metadata stored in a map data structure. This segmentation allows the system to identify and decompress only the specific chunk containing the requested data portion, rather than decompressing all preceding data. The map stores chunk boundaries and compression information, enabling direct navigation to the target chunk.
Solution Approach 2:
The patent performs preliminary compression of data chunks and stores metadata about each chunk in a map data structure before retrieval operations. This preliminary action includes calculating and storing chunk sizes, compression ratios, and position information, which enables fast identification of the target chunk during ranged reads without requiring full decompression of preceding data.
2Ease of operation
If all preceding data is decompressed to access a specific portion of compressed data, then the desired data portion can be accessed, but time consumption increases
Solution Approach 1:
The patent segments the compressed data object into discrete chunks with stored metadata in a map data structure. This segmentation enables the system to directly locate and decompress only the specific chunk containing the requested data, eliminating the time-wasting sequential decompression of all preceding chunks. The map provides O(1) or O(log n) access to chunk position information.
Solution Approach 2:
The patent creates and maintains a map data structure that copies essential metadata about each compressed chunk (positions, sizes, compression information) separately from the actual compressed data. This copied metadata structure enables rapid lookup and direct access to chunk boundaries without traversing or decompressing the actual data, significantly reducing access time for ranged reads.
3Productivity
If a map data structure is used to track compressed chunk sizes, then decompression can start at the requested position, but memory usage increases
Solution Approach 1:
The patent implements a map data structure that stores only the essential local metadata needed for chunk identification (compressed sizes, positions) rather than complete compression information for all chunks. This local quality approach keeps the map compact and memory-efficient while providing sufficient information to navigate to and decompress the target chunk. The map stores minimal necessary data: chunk start positions and compressed sizes.
4Productivity
If the map data structure is compacted to fit in memory, then decompression can initiate at the requested offset, but the compaction process adds complexity
Solution Approach 1:
The patent merges the map data structure with the compressed data object storage, storing the map either inline with the compressed data or in a tightly coupled format. This merging eliminates the need for separate complex compaction processes by integrating the metadata storage directly into the data flow. The map is constructed incrementally during compression and stored in a compact format that fits efficiently in memory without requiring post-processing compaction operations.
Data Source
AI summary
In some examples, a computer system may receive a plurality of chunks of data of a data object. The system may compress the plurality of chunks of data to obtain a plurality of compressed chunks, and may determine whether the plurality of compressed chunks together are less than a threshold size. Based on determining that the plurality of compressed chunks together are less than the threshold size, the system may add, to respective entries in a map data structure, respective sizes of the plurality of compressed chunks. In addition, the system may compact the map data structure by combining values in at least two of the respective entries, and may store the plurality of compressed chunks and the compacted map data structure.