Self-Checking Compression With Hidden Parity for Early Error Detection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current compression schemes lack effective error detection mechanisms, leading to silent data corruption and inefficient bandwidth utilization due to the inability to detect errors until the end of data transfer, which can result in significant delays and data loss.

Innovation Solution

Implementing a novel compression and decompression scheme that incorporates hidden parity information within the LZ77 compression format, allowing for continuous error detection during data transfer without relying on higher-level framing layers, enabling early detection and localization of errors.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If a single checksum is defined over the entire compressed stream, then data integrity can be verified, but error detection is delayed until the end of decompression and provides no information about where errors occurred

Engineering Contradiction:

Improvedata integrity verificationVSAvoiderror detection delay

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent divides the single checksum verification into multiple distributed checksums placed throughout the compressed stream. Each checksum covers a specific segment of the compressed data, allowing error detection to occur at multiple points during decompression rather than waiting until the end. This segmentation enables early error detection and localization without compromising overall data integrity verification.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent places checksums at predetermined intervals within the compressed stream before decompression is complete. This preliminary placement of verification points allows the decompressor to detect and report errors as they are encountered during the decompression process, rather than waiting until the final checksum verification at the end of the entire stream.

Inventive Principle:

Principle #10Preliminary action

2Reliability

If the entire compressed stream is decompressed before verification, then complete data can be checked, but a huge amount of data must be buffered which is inefficient for large files

Engineering Contradiction:

Improvedata verification completenessVSAvoidbuffer memory requirement

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The patent segments the compressed stream into multiple sections, each terminated by a checksum. The decompressor processes and verifies each segment independently as it decompresses, rather than buffering the entire stream. This segmentation allows verification to occur in smaller chunks, dramatically reducing the buffer memory requirement while maintaining complete verification coverage throughout the entire file.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent enables continuous verification during the decompression process by placing checksums at regular intervals. As each segment is decompressed, its checksum is immediately verified, allowing the process to continue to the next segment without pausing to buffer large amounts of data. This continuous action maintains verification completeness while minimizing memory usage.

Inventive Principle:

Principle #20Continuity of useful action

3Productivity

If no error detection is implemented in the compression scheme, then bandwidth is used efficiently, but silent data corruption occurs without detection

Engineering Contradiction:

Improvebandwidth efficiencyVSAvoiddata corruption detection

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The patent merges error detection functionality directly into the compression scheme by embedding checksums within the compressed stream structure. Rather than adding separate error detection layers that would increase overhead, the checksums are integrated into the compression format itself, allowing error detection to occur alongside compression and decompression operations without significantly impacting bandwidth efficiency.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The compressed stream structure provides its own error detection capability through embedded checksums. The system does not require external error detection mechanisms or higher-level framing layers to detect corruption. The compression format itself is self-sufficient, containing within its structure the means to verify data integrity and detect corruption during the decompression process.

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS11483009B2Self-checking compression

Publication Date: 2022.10.25 INTEL CORP

AI summary

Methods, apparatus, systems, and software for implementing self-checking compression. A byte stream is encoded to generate tokens and selected tokens are encoded with hidden parity information in a compressed byte stream that may be stored for later streaming or streamed to a receiver. As the compressed byte stream is received, it is decompressed, with the hidden parity information being decoded and used to detect for errors in the decompressed data, enabling errors to be detected on-the-fly rather than waiting to perform a checksum over an entire received file. In one embodiment the byte stream is encoded using a Lempel-Ziv 77 (LZ77)-based encoding process to generate a sequence of tokens including literals and references, with all or selected references encoded with hidden parity information in a compressed byte stream having a standard format such as DEFLATE or Zstandard. The hidden parity information is encoded such that the compressed byte stream may be decompressed without parity checks using standard DEFLATE or Zstandard decompression schemes. Dictionary coders such as LZ78 and LZW may also be used.