Content-Agnostic File Indexing via Data Permutation Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current file referencing and data compression techniques require knowledge of the data type, limiting their efficiency in reducing storage and transmission requirements.
Innovation Solution
A content-agnostic method that calculates permutations of data based on its length, allowing for efficient indexing and compression by storing or transmitting the length and index instead of the data itself, enabling flexible and reduced data handling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional file referencing and compression techniques are used, then data type knowledge is required for efficient indexing, but this limits the system's ability to handle any content type
Solution Approach 1:
The patent creates a universal indexing system that works with any data type by treating all data as sequences of tokens. The tokenizer converts different content types (text, binary, images, videos) into a common token representation, allowing the same indexing algorithms to be applied universally without requiring type-specific handling logic.
Solution Approach 2:
The system changes the parameter representation of data by converting raw data into token sequences with associated metadata (token IDs, frequencies, positions). This parameter transformation allows diverse content types to be represented in a unified format that can be efficiently indexed and compressed using the same methods.
2Quantity of substance
If data is persisted in traditional formats, then complete data must be stored, but this increases storage and transmission requirements
Solution Approach 1:
The patent extracts only the essential indexing information (token sequences, frequencies, positions) from the complete data, storing this compressed representation instead of the full data. The original data can be reconstructed on-demand by retrieving the indexed tokens and regenerating them, significantly reducing storage requirements while maintaining data accessibility.
Solution Approach 2:
The system creates a compressed copy of the data in the form of an index structure containing token references rather than the actual data content. This index copy is sufficient for search and retrieval operations, eliminating the need to store and transmit complete data copies for every access operation.
3Productivity
If content-specific compression methods are used, then compression efficiency is improved for specific types, but the system cannot handle diverse content types effectively
Solution Approach 1:
The patent develops a universal compression approach that works across all content types by compressing the token sequences and metadata rather than the raw data directly. The compression algorithms operate on the standardized token representation, achieving efficient compression for text, binary files, images, and videos without requiring separate compression methods for each type.
Data Source
AI summary
A computer-implemented method for content-agnostic referencing of a binary data file, the method comprising: pregenerating a table of all permutations of data of a particular length, determining a length of the binary data file, the length comprising the number of bits of the binary data file; chunking the binary data into chunks of data of a smaller length; for each chunk, determining if the chunk is in the pregenerated table, and if so using that chunks index in the pregenerated table, and otherwise chunking the data again until the sub-chunks are located in the pregenerated table, and using the number of chunks and associated indices to indicate the binary data file.