Compressed Data Sub-Block Indexing for Random Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional compression formats for big data stores, such as Deflate, offer poor compression ratios and do not allow for random access to compressed data, making them costly and resource-intensive, while also risking data corruption during the compression process.
Innovation Solution
The implementation of a data compression technique that generates compressed data in discrete sub-blocks with a block index and checksum, enabling random access and verification of data integrity, allowing for decompression of only the necessary sub-blocks and ensuring data accuracy through checksum matching.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional compression formats (e.g., Deflate) are used, then compression ratio is improved, but random access capability deteriorates
Solution Approach 1:
The patent divides compressed data into discrete sub-blocks with individual checksums and creates a block index that maps sub-block positions. This segmentation allows random access to specific sub-blocks without decompressing the entire data set, resolving the contradiction between achieving high compression ratios and maintaining random access capability.
2Ease of manufacture
If conventional compression formats are used, then implementation simplicity is improved, but data integrity verification deteriorates
Solution Approach 1:
The patent generates checksums for each sub-block during the compression process and stores them in a block index. This preliminary verification action allows data integrity to be checked before decompression, maintaining implementation simplicity while significantly improving data integrity verification capability.
3Ease of operation
If decompression from start point to point of interest is performed, then random access is enabled, but resource consumption deteriorates
Solution Approach 1:
The patent extracts and stores metadata including block indices and checksums separately from the compressed data. This extraction enables direct access to specific sub-blocks through the index without decompressing unnecessary portions, enabling random access while minimizing resource consumption.
4Quantity of substance
If compression processes are performed, then data size is reduced, but data corruption risk increases
Solution Approach 1:
The patent implements checksum verification as a feedback mechanism during and after compression. Checksums are generated for each sub-block and used to verify data integrity, providing continuous feedback that detects corruption while maintaining data size reduction through compression.
Data Source
AI summary
A processing device comprising compression circuitry to: determine a compression configuration to compress source data; generate a checksum of the source data in an uncompressed state; compress the source data into at least one block based on the compression configuration, wherein the at least one block comprises: a plurality of sub-blocks, wherein the plurality of sub-block includes a predetermined size; a block header corresponding to the plurality of sub-blocks; and decompression circuitry coupled to the compression circuitry, wherein the decompression circuitry to: while not outputting a decompressed data stream of the source data: generate index information corresponding to the plurality of sub-blocks; in response to generating the index information, generate a checksum of the compressed source data associated with the plurality of sub-blocks; and determine whether the checksum of the source data in the uncompressed format matches the checksum of the compressed source data.