Compressed Data Chunk Mapping for Fast Ranged Reads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for accessing compressed data through ranged reads require decompressing all preceding data, leading to computational inefficiency and high time consumption, as they do not enable decompression to start at a specific requested position without determining the compression of preceding data.
Innovation Solution
A map data structure is generated to associate compressed and uncompressed positions, allowing decompression to initiate at the requested range without decompressing the entire object, by correlating original uncompressed object positions with offsets in the compressed data, and using chunk sizes to optimize map size and compaction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If all preceding data is decompressed to enable access to a requested portion, then the desired data can be accessed, but computational overhead and time consumption increase significantly
Solution Approach 1:
The patent divides the compressed data object into multiple chunks, where each chunk is independently compressed and can be independently decompressed. This segmentation allows the system to decompress only the specific chunk containing the requested data portion, rather than decompressing all preceding data. The chunk structure enables selective access to different portions of the compressed object without processing the entire dataset.
Solution Approach 2:
The patent performs preliminary compression of data into independently manageable chunks before storage. During this preliminary action, each chunk is compressed separately with its own compression parameters and metadata. This pre-processing enables rapid random access during retrieval operations, as the system can directly locate and decompress only the relevant chunk based on the requested position, without needing to process preceding chunks.
2Quantity of substance
If conventional compression methods are used, then storage space is optimized, but ranged read performance deteriorates due to full decompression requirements
Solution Approach 1:
The patent implements a chunk-based compression architecture where the data object is divided into multiple independently compressed segments. Each chunk maintains its own compression state and metadata, allowing selective decompression of individual chunks for ranged read operations. This segmentation preserves storage optimization benefits while enabling efficient partial data access without full decompression overhead.
Solution Approach 2:
The patent introduces an intermediary index structure that maps logical positions in the original data object to physical positions in the compressed chunks. This intermediary layer enables the system to quickly determine which specific chunk contains the requested data portion and retrieve only that chunk for decompression, significantly improving ranged read performance while maintaining compression efficiency.
3Productivity
If a map data structure is generated to enable selective decompression, then ranged read performance improves, but memory requirements and map maintenance overhead increase
Solution Approach 1:
The patent segments the map data structure to correspond with the chunk segmentation of the compressed data. Each chunk has an associated simplified map entry containing only the necessary metadata (compression parameters, size, position) for that specific chunk. This segmented approach reduces the complexity of the overall map structure compared to a single comprehensive map, as each chunk's map entry is independent and can be processed separately.
Solution Approach 2:
The patent extracts only the essential metadata information needed for selective decompression into the map structure, omitting unnecessary details. The map contains specifically extracted elements such as chunk boundaries, compression algorithms used, and position mappings, while excluding redundant information. This extraction principle minimizes map size and complexity while maintaining the functionality required for efficient ranged read operations.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
In some examples, a computer system may receive a plurality of chunks of data of a data object. The system may compress the plurality of chunks of data to obtain a plurality of compressed chunks, and may determine whether the plurality of compressed chunks together are less than a threshold size. Based on determining that the plurality of compressed chunks together are less than the threshold size, the system may add, to respective entries in a map data structure, respective sizes of the plurality of compressed chunks. In addition, the system may compact the map data structure by combining values in at least two of the respective entries, and may store the plurality of compressed chunks and the compacted map data structure.