Prime Data Sieve for Lossless Reduction and Random Data Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data compression methods are inefficient in handling large and extremely large datasets, as they can only exploit redundancy within a local window and are not suited for random access or high-speed data ingestion and retrieval, leading to limitations in data storage and processing capabilities.
Innovation Solution
The use of a Prime Data Sieve to factorize input data into Prime Data Elements and Derivative Elements, allowing for content-associative retrieval and storage, enabling efficient lossless data reduction across the entire dataset by organizing and storing data in a tree-like structure that facilitates quick access and derivation of elements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional data compression methods are used to reduce data footprint, then storage space is reduced, but the ability to perform random access and high-speed retrieval is degraded
Solution Approach 1:
The patent segments data into fixed-size chunks and organizes them in a tree structure with metadata blocks containing indexes. This segmentation allows random access to specific data chunks without decompressing entire datasets, resolving the contradiction between compression and retrieval speed by enabling selective access to compressed data segments.
Solution Approach 2:
The patent introduces metadata blocks as intermediaries between compressed data and retrieval operations. These metadata blocks contain indexes and descriptors that enable fast lookup and direct access to compressed data chunks, allowing random access without full decompression and thus maintaining both compression benefits and retrieval speed.
2Quantity of substance
If data is organized in a storage hierarchy with multiple tiers, then storage density is improved, but access latency increases for lower tiers
Solution Approach 1:
The patent performs preliminary organization of data into fixed-size chunks with associated metadata blocks containing indexes and descriptors. This preliminary structuring enables direct navigation to specific data locations without sequential scanning, reducing access latency when retrieving data from dense storage tiers while maintaining high storage density.
3Quantity of substance
If universal lossless data reduction techniques are applied to diverse data formats, then data footprint is reduced, but processing complexity increases
Solution Approach 1:
The patent changes the parameter of data organization by imposing a fixed-size chunk structure on diverse data formats. This parameter change enables uniform processing and compression of different data types through the same tree-based organization and compression algorithms, reducing data footprint while managing processing complexity through standardized handling.
Data Source
Figure 1A
Figure 1B
Figure 1C
AI summary
Input data can be losslessly reduced by using a data structure that organizes prime data elements based on their contents. Alternatively, the data structure can organize prime data elements based on the contents of a name that is derived from the prime data elements. Specifically, video data can be losslessly reduced by (1) using the data structure to identify a set of prime data elements, and (2) using the set of prime data elements to losslessly reduce intra-frames. The input data can be dynamically partitioned based on the memory usage of components of the data structure. Parcels can be created based on the partitions to facilitate archiving and movement of the data. The losslessly reduced data can be stored using a set of distilled files and a set of prime data element files.