Compressed Column Data Consistency Checks for Corruption Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional database systems fail to proactively detect corruption in binary compressed data structures, leading to incorrect query results, performance degradation, and resource-intensive recovery processes.
Innovation Solution
Implement a system that performs consistency checks on compressed data structures by determining if they satisfy mathematical criteria specific to the compression algorithm used, triggered by separate instructions and executed during write transactions or periodically, using metadata to identify the compression algorithm and applying algorithm-specific checks to detect inconsistencies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is compressed to reduce memory usage, then storage efficiency is improved, but data corruption risk increases and detection capability deteriorates
Solution Approach 1:
The patent applies preliminary action by embedding checksums and compression metadata into the compressed data structure at the time of compression, rather than performing detection after corruption occurs. This allows the system to proactively verify data integrity before queries are executed, preventing undetected corruption from affecting query results.
Solution Approach 2:
The patent introduces checksums and metadata as intermediary elements that mediate between the compressed data and the verification process. These intermediaries enable indirect verification of data integrity without requiring full decompression, thus maintaining both storage efficiency and detection capability.
2Reliability
If corruption detection is performed proactively, then data reliability is improved, but system complexity and processing overhead increase
Solution Approach 1:
The patent extracts the verification function from the compression process by separating checksum calculation and validation into distinct operations. The compression algorithm remains unchanged, while a separate verification mechanism uses the embedded metadata to detect corruption, reducing overall system complexity.
Solution Approach 2:
The compressed data structure performs self-verification through embedded checksums and metadata that enable the data to validate its own integrity without requiring external verification systems. This self-service approach reduces the complexity of external detection mechanisms.
3Ease of repair
If undo/redo logs are maintained for data recovery, then data restoration capability is improved, but resource consumption and recovery time increase
Solution Approach 1:
The patent prepares verification data (checksums and metadata) in advance during the compression process, so that when corruption is detected, the system can immediately identify and restore only the affected data segments without needing to maintain extensive undo/redo logs for entire datasets.
Solution Approach 2:
The patent applies local quality by enabling targeted recovery of only the corrupted data segments rather than requiring full dataset restoration. The embedded metadata allows precise identification of corruption locations, so recovery resources are concentrated only where needed.
Data Source
AI summary
Systems and methods include reception of an instruction to perform a consistency check on compressed column data. In response to the instruction, a compression algorithm applied to uncompressed column data to generate the compressed column data is determined, one or more consistency checks associated with the compression algorithm are determined, wherein a first one or more consistency checks associated with a first compression algorithm are different from a second one or more consistency checks associated with a second compression algorithm, the one or more consistency checks are executed on the compressed column data, and, if the one or more consistency checks are not satisfied, a notification is transmitted to a user.