Compressed Data Sub-Blocks for Random Access and Checksum Verification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional compression formats for big data stores, such as Deflate, offer poor compression ratios and hinder random access, making them costly and resource-intensive, while also risking data corruption during the compression process.
Innovation Solution
The implementation of a data compression technique that generates compressed data in discrete sub-blocks with associated checksums, allowing for random access and verification of data integrity, ensuring that the compressed data can be successfully decompressed without corruption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If conventional compression formats (e.g., Deflate) are used, then random access is allowed to some degree, but compression ratios are poor and resource consumption is high
Solution Approach 1:
The patent divides compressed data into discrete sub-blocks with individual checksums, allowing random access to specific sub-blocks without decompressing the entire data set. This segmentation enables efficient point-of-interest access while maintaining high compression ratios through sophisticated compression algorithms.
2Productivity
If sophisticated compression algorithms are used, then compression ratios improve, but random access becomes prohibitively costly
Solution Approach 1:
By segmenting compressed data into independently checksummed sub-blocks, the patent enables random access to specific segments without processing the entire compressed data set, thereby reducing the cost of random access operations while maintaining high compression ratios.
Solution Approach 2:
The patent pre-computes and stores checksums for each sub-block during the compression phase. This preliminary action enables rapid verification and access to specific sub-blocks during random access operations, eliminating the need to decompress and verify the entire data set.
3Productivity
If data is compressed without verification, then processing is faster, but data corruption risk increases
Solution Approach 1:
The patent performs checksum verification during the compression phase itself, rather than after compression is complete. This preliminary verification ensures data integrity is confirmed while the data is still being processed, allowing for immediate detection and correction of compression errors without requiring re-processing.
Solution Approach 2:
By dividing data into sub-blocks with individual checksums, the patent isolates verification to small segments rather than the entire data set. This segmented verification maintains high compression speed while ensuring data integrity through targeted checksum validation of each sub-block.
Data Source
AI summary
A processing device comprising compression circuitry to: determine a compression configuration to compress source data; generate a checksum of the source data in an uncompressed state; compress the source data into at least one block based on the compression configuration, wherein the at least one block comprises: a plurality of sub-blocks, wherein the plurality of sub-block includes a predetermined size; a block header corresponding to the plurality of sub-blocks; and decompression circuitry coupled to the compression circuitry, wherein the decompression circuitry to: while not outputting a decompressed data stream of the source data: generate index information corresponding to the plurality of sub-blocks; in response to generating the index information, generate a checksum of the compressed source data associated with the plurality of sub-blocks; and determine whether the checksum of the source data in the uncompressed format matches the checksum of the compressed source data.