Reference Codebook Deduplication for Secure Compacted Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The rapid growth of data storage demand exceeds the capacity to store it, and existing data compression and physical storage solutions are inadequate, while transmission bandwidth and data security are becoming bottlenecks, especially with the rise of quantum computing.
Innovation Solution
A system and method for secure deduplication of compacted data using a data deconstruction engine, data reconstruction engine, library manager, and reference codebook that performs simultaneous compaction and deduplication by comparing sourceblocks against a reference codebook to identify and store unique data, ensuring duplicates are not stored.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data compression is used to increase storage capacity, then storage efficiency improves, but data security deteriorates due to encryption vulnerabilities
Solution Approach 1:
The patent segments data into fixed-size sourceblocks before encoding. Each sourceblock is independently processed and referenced in the codebook, creating discrete security units that can be individually protected and verified, thereby maintaining security while enabling compression.
Solution Approach 2:
The system performs preliminary hashing and codebook generation before actual data storage. By pre-processing data into sourceblocks and creating reference codes in advance, the system establishes security protocols and compression structures beforehand, ensuring both security and efficiency are built into the storage architecture from the start.
2Quantity of substance
If physical storage capacity is increased to meet demand, then storage availability improves, but transmission bandwidth requirements worsen
Solution Approach 1:
The patent creates a compressed representation (codebook with reference codes) that copies only essential information from the original data. Instead of transmitting or storing full data sets, the system transmits compacted codebooks containing reference codes that point to sourceblocks, dramatically reducing bandwidth requirements while maintaining storage capacity.
Solution Approach 2:
The system changes the parameter of data representation from full-data storage to reference-based storage. By transforming data into a compacted form with reference codes and sourceblock identifiers, the system reduces the amount of data that needs to be transmitted while preserving the ability to retrieve and verify original data content.
3Quantity of substance
If data deduplication is implemented to reduce storage needs, then storage efficiency improves, but system complexity worsens
Solution Approach 1:
The codebook structure enables self-service deduplication where the system automatically identifies and references duplicate sourceblocks without requiring complex external management. The reference code mechanism allows the data structure itself to track and eliminate duplicates, reducing the need for additional complexity in deduplication management.
Data Source
AI summary
A system and methods for secure deduplication of compacted data comprising a data deconstruction engine, a data reconstruction engine, a library manager, a reference codebook, and a codeword storage which performs simultaneous compaction and deduplication of data sets. A data set may be comprised of one or more sourcepackets which may be optimally deconstructed into a plurality of sourceblocks and wherein each sourceblock may be compared against a reference codebook that contains key-value pairs of a sourceblock and its associated reference code in order to determine if a received sourceblock is a duplicate of data already stored within the reference codebook. Non-duplicate sourceblocks can have a reference code algorithmically created and stored in the reference codebook, thereby ensuring that when a duplicate sourceblock is received, it will not be stored as duplicated data.