Prime Data Sieve Indexing for Searchable Lossless Data Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data compression methods are inefficient in handling large and extremely large datasets, as they can only exploit redundancy within a local window size that fits in processor cache or memory, and are not suited for random access or multidimensional search, leading to limitations in data ingestion and retrieval rates.
Innovation Solution
The use of a Prime Data Sieve to factorize input data into Prime Data Elements and Derivative Elements, allowing for content-associative search and retrieval by navigating a tree data structure based on element names, and generating losslessly reduced representations using Reconstitution Programs, enabling efficient data reduction and retrieval across large datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in traditional compression formats, then data footprint is reduced, but search and retrieval operations become inefficient
Solution Approach 1:
The patent segments data into fixed-size chunks and creates hierarchical indexes (level-0, level-1, level-2 indexes) that organize these segments. This segmentation allows the system to reduce data footprint through compression while maintaining efficient search capabilities by navigating through organized index structures rather than scanning compressed data directly.
Solution Approach 2:
The patent introduces hierarchical index structures as intermediaries between the stored data and search operations. These indexes (particularly level-2 indexes containing minimum and maximum values) enable the system to filter and navigate data efficiently without decompressing entire datasets, thus maintaining both reduced footprint and high retrieval rates.
2Quantity of substance
If compression window size is increased to exploit more redundancy, then data reduction ratio improves, but memory requirements and processing complexity increase
Solution Approach 1:
The patent divides data into fixed-size chunks for compression processing. This segmentation allows the system to apply compression algorithms to manageable units while maintaining the ability to exploit redundancy across chunk boundaries through the hierarchical index structure, thus achieving high data reduction without requiring excessively large in-memory windows.
Solution Approach 2:
The patent transitions from single-dimensional compression (linear scanning) to multi-dimensional organization through hierarchical indexes. Level-0 indexes organize chunks sequentially, level-1 indexes group them by ranges, and level-2 indexes provide global minimum-maximum filtering. This dimensional organization enables efficient redundancy exploitation without increasing processing complexity.
3Quantity of substance
If data is compressed to reduce storage costs, then storage capacity requirements decrease, but random access capability is lost
Solution Approach 1:
The patent segments data into fixed-size chunks that are independently compressible and indexable. This segmentation enables random access to specific chunks through the hierarchical index structure without requiring decomposition of the entire dataset, thus maintaining random access capability while achieving storage reduction.
Solution Approach 2:
The patent performs preliminary organization of data into chunks and creates hierarchical indexes before compression and storage. This preliminary action enables efficient random access operations later, as the index structures allow direct navigation to desired data regions without decompressing unrelated portions.
4Device complexity
If traditional compression algorithms are used, then implementation simplicity is maintained, but they cannot handle extremely large datasets efficiently
Solution Approach 1:
The patent segments extremely large datasets into fixed-size chunks that can be processed independently and in parallel. This segmentation maintains implementation simplicity by using standard compression algorithms on each chunk while dramatically improving data ingestion rate through parallel processing and efficient index-based navigation.
Solution Approach 2:
The patent adds hierarchical indexing as an additional organizational dimension beyond simple sequential processing. This multi-level index structure (level-0, level-1, level-2) enables efficient handling of extremely large datasets by allowing parallel processing of chunks and rapid navigation through the data hierarchy, thus improving productivity without sacrificing implementation simplicity.
Data Source
Figure 1A
Figure 1B
Figure 1C
AI summary
This disclosure relates to multidimensional search and retrieval, and keyword-based search and retrieval on losslessly reduced data by organizing prime data elements (the data structure that organizes the prime data elements is called the prime data sieve) using components of a structure of an input dataset so that searches can be performed on the losslessly reduced data based on the one or more components of the structure. For every prime data element, the prime data sieve can retain a reverse reference to the losslessly reduced representation of each chunk that refers to the prime data element. Some embodiments create and maintain a keyword index for losslessly reduced data. Some embodiments losslessly reduce input data using a distributed system.