Content Indexing With Word Pointers for Storage Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data storage requirements are increasing faster than storage technology capabilities, and existing compression techniques slow down data retrieval while conflicting with the need for indexing, which is essential for search and retrieval efficiency.
Innovation Solution
A method that indexes content by finding semantic units in files, adding them to an index if not already present, and replacing instances with pointers, thereby reducing overall storage needs by combining indexing and data reduction in a single-pass process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data compression techniques are applied to reduce storage requirements, then storage efficiency is improved, but data retrieval speed deteriorates due to decompression overhead
Solution Approach 1:
The patent pre-processes data during ingestion to create an index structure that stores semantic units and their locations. This preliminary action eliminates the need for decompression during retrieval, as the index directly references original data positions, thus improving retrieval speed while maintaining compression benefits.
Solution Approach 2:
The patent introduces an index as an intermediary structure between compressed data and retrieval operations. The index contains semantic units and pointers to their locations in the original data, allowing fast lookup without full decompression. This intermediary resolves the contradiction by enabling quick access paths that bypass the decompression bottleneck.
2Productivity
If indexing is performed to improve search and retrieval efficiency, then productivity is improved, but storage requirements increase due to index size
Solution Approach 1:
The patent merges the index with the compressed data storage system, where the index structure is integrated into the overall storage architecture. By combining indexing with compression, the system achieves both goals simultaneously - the index enables efficient search while the compression reduces overall storage needs, offsetting the index overhead.
Solution Approach 2:
The patent creates a multi-functional index structure that serves both as a search index and as part of the compression scheme. The index stores semantic units that can be used for both retrieval operations and compression dictionary functions, making the storage system more efficient overall by having the index perform multiple roles rather than requiring separate structures.
3Quantity of substance
If compression is applied to reduce data size, then storage efficiency is improved, but data becomes unreadable and therefore not indexible
Solution Approach 1:
The patent performs preliminary indexing during the data ingestion phase, before compression is fully applied or while the data is still in a processable state. By creating the index structure in advance when data is more accessible, the system maintains indexability while still achieving compression benefits in the final stored format.
Solution Approach 2:
The patent uses semantic units as intermediaries between the compressed data and the indexing system. These semantic units serve as a bridge that allows the index to reference compressed data without requiring the full data to be readable. The semantic units capture the essential meaning and structure needed for indexing while working within the compressed format.
Data Source
AI summary
Data storage is improved by combining content indexing and data reduction in text-containing files by using common word elimination. Raw data is processed by finding words in selected files, creating an index of found words, and replacing the words in the raw data with pointers to the corresponding words in the index. Each word appears only once in the index. Consequently, the index is relatively small and the procedure is completely reversible. In particular, the index is small relative to other methods because the data is transformed in place, and the transformed data and index are used together to capture the total information about the data.