Random Access for Chunked Compressed Files
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud-based storage environments face inefficiencies in providing random access to large, compressed files, as decompressing from the beginning is required, leading to performance and cost issues, especially when files are split into parts and compressed separately.
Innovation Solution
The system segments files into parts with indices containing frame numbers, offsets, lengths, and original lengths, allowing for efficient retrieval of compressed frames by mapping ranges using these indices, and decompressing only the requested portions, while also handling multipart files by mapping ranges across parts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If the whole file is decompressed to return the requested portion, then random access to compressed files is enabled, but computational overhead and cost increase significantly
Solution Approach 1:
The patent divides large files into multiple chunks or segments before compression. Each chunk is independently compressed and stored with metadata information. This segmentation allows the system to retrieve and decompress only the specific chunks needed for a given range request, rather than decompressing the entire file, thus reducing computational overhead while maintaining random access capability.
2Quantity of substance
If large files are split into parts and compressed separately, then storage efficiency is improved, but mapping requested ranges to compressed data becomes complex
Solution Approach 1:
The patent pre-computes and stores metadata for each chunk, including the original byte ranges, compressed sizes, and cumulative offset information. This preliminary action creates an index structure that enables efficient mapping of requested byte ranges to specific chunks and positions within chunks, simplifying the retrieval process despite file segmentation.
3Volume of stationary object
If files are compressed to reduce storage space, then storage costs decrease, but random access performance deteriorates
Solution Approach 1:
The patent segments files into chunks that are compressed individually. The metadata stored for each chunk contains information about its original size, compressed size, and position. This segmentation combined with metadata indexing allows the system to quickly locate and decompress only the specific chunks needed for a range request, maintaining fast access performance while achieving storage compression.
4Ease of operation
If the requested portion is retrieved from a compressed file, then data access is enabled, but the entire file must be decompressed first
Solution Approach 1:
The patent divides the file into independently compressed chunks. When a range request is received, the system uses the metadata to identify which specific chunks contain the requested data, retrieves only those chunks, and decompresses them locally. This eliminates the need to decompress the entire file, significantly reducing decompression time and enabling efficient random access.
Data Source
AI summary
Embodiments of the disclosure provide systems and methods for providing random access to segmented and encrypted or compressed data stored in a repository. Retrieving at least a portion of a file stored in a repository can comprise storing a plurality of files in the repository. A request to retrieve at least a portion of one of the plurality of files can be received and object metadata for the requested one of the plurality of files can be obtained. A determination can be made based on the obtained metadata as to whether the requested one of the plurality of files is a multipart file. In response to determining the requested one of the plurality of files is not a multipart file, a single file retrieval process performing and in response to determining the requested one or the plurality of files is a multipart file, a multipart retrieval process can be performed.