Prime Data Element Compression for Random-Access Data Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression techniques are limited in their ability to efficiently discover and exploit redundancy across large and extremely large datasets, leading to unacceptably slow ingest and retrieval rates, especially when the redundant data is spatially or temporally separated by multiple terabytes, petabytes, or exabytes, and they are not suited for random access of data.
Innovation Solution
The method involves breaking down input data into Prime Data Elements, which are stored in a Prime Data Sieve, and deriving Derivative Elements through transformations specified in a Reconstitution Program, allowing for a lossless data reduction by referencing these elements, thus enabling efficient data reduction and retrieval across large datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If existing data compression techniques are used, then data storage capacity is improved, but ingest and retrieval rates become unacceptably slow
Solution Approach 1:
The patent segments data into fixed-size chunks and processes them independently through compression and decompression operations. This segmentation enables parallel processing of multiple chunks simultaneously, improving ingest and retrieval rates while maintaining compression efficiency for large datasets
Solution Approach 2:
The patent performs preliminary compression of data chunks before storage, and preliminary allocation of decompression resources before retrieval operations. This preliminary action prepares data in advance, reducing the actual ingest and retrieval time when data needs to be accessed
2Quantity of substance
If redundancy is searched across large datasets separated by terabytes or petabytes, then data reduction is improved, but processing time becomes unacceptably slow
Solution Approach 1:
The patent divides large datasets into manageable chunks that can be processed independently. By segmenting the search space for redundancy, the system can process multiple segments in parallel, finding redundancy across terabytes or petabytes of data without sequentially scanning everything, thus reducing processing time
Solution Approach 2:
The patent performs partial redundancy checks on data chunks, examining only the necessary portions of data to identify compressible patterns. This partial action approach finds sufficient redundancy to achieve data reduction without performing exhaustive searches that would be time-consuming across extremely large datasets
3Quantity of substance
If traditional compression methods are used, then data footprint is reduced, but random access capability is lost
Solution Approach 1:
The patent segments compressed data into fixed-size chunks with metadata that enables independent access to individual chunks. This segmentation structure allows random access to specific data portions without requiring decompression of the entire dataset, maintaining random access capability while achieving data footprint reduction through compression
Solution Approach 2:
The patent creates compressed copies of data chunks that can be stored efficiently while maintaining the ability to reconstruct original data when needed. These compressed copies reduce data footprint but include mechanisms to retrieve and decompress specific chunks for random access operations
Data Source
AI summary
A data structure may be navigated based on a data chunk to identify a prime data element, where the data structure may organize a set of prime data elements based on contents of the set of prime data elements. A derivation may be determined which derives the data chunk from the prime data element. A distance metric of the derivation may be compared with a distance threshold to obtain a comparison result. A first representation of the data chunk may be determined based on the comparison result, where the first representation of the data chunk may include a reference to the prime data element and a second representation of the derivation. The first representation of the data chunk may be stored on a storage device.