Uncompressed Metadata Chunks for Compressed Data Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current computing systems require extensive time and resources for decoding and decompressing large sets of digital data, making it inefficient to search and access desired data.
Innovation Solution
Embedding uncompressed metadata chunks within data files that include compressed data, allowing for efficient searching by indicating the ranges of compressed data that contain the desired values, thereby reducing the need for decompression and minimizing resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in compressed format to reduce storage space, then storage efficiency is improved, but decoding and decompression time increases
Solution Approach 1:
The patent pre-computes and stores metadata information (min/max values, null value counts, distinct value counts) during the data compression phase. This preliminary action allows the system to quickly determine whether a compressed data chunk contains relevant data without performing full decompression, thus resolving the contradiction between storage efficiency and access time.
Solution Approach 2:
The patent introduces metadata chunks as an intermediary layer between the compressed data chunks and the query processing system. These metadata chunks contain summary statistics that enable fast filtering and decision-making about which compressed data chunks need to be decompressed, thereby reducing overall decoding time while maintaining storage efficiency.
2Ease of operation
If all compressed data is decompressed for searching to ensure complete data access, then data accessibility is improved, but CPU resource consumption increases
Solution Approach 1:
The patent applies partial action by decompressing only the necessary portions of data that are likely to contain the desired information. Using metadata filters, the system identifies and decompresses only relevant compressed data chunks rather than all data, thus maintaining data accessibility while significantly reducing CPU resource consumption.
Solution Approach 2:
The patent segments the data storage into multiple compressed data chunks, each with its own metadata chunk. This segmentation allows the system to independently evaluate and process only those segments that are relevant to the query, avoiding the need to decompress and process the entire dataset, thereby reducing CPU usage while preserving complete data access capability.
3Quantity of substance
If metadata is stored in compressed format to save space, then storage efficiency is improved, but search speed decreases
Solution Approach 1:
The patent applies local quality by storing metadata in uncompressed format while keeping the main data chunks compressed. The metadata, being small in size and requiring frequent access for filtering, benefits from uncompressed storage that enables fast reading and processing, while the large data chunks remain compressed for storage efficiency. This resolves the contradiction by optimizing each component according to its specific access patterns and size requirements.
Data Source
AI summary
A data processing method provides an improvement in speed of obtaining, in response to a computer search query from one or more computing devices, a particular compressed data chunk that may include a particular data value that is digitally stored in a large data file. The method comprises creating and digitally storing a digital data file comprising a first set of uncompressed metadata chunks; a second set of uncompressed metadata chunks; and a set of compressed data chunks; each chunk of the first and second sets of uncompressed metadata chunks comprising a set of bits; the set of compressed data chunks comprising a first compressed data chunk; the first compressed data chunk comprising a first set of values; the first set of uncompressed metadata chunks spaced evenly across the data file; the second set of uncompressed metadata chunks spaced evenly between a first metadata chunk of the first set of uncompressed metadata chunks and a second metadata chunk of the first set of uncompressed metadata chunks; the first metadata chunk comprising a first set of bits. The method further comprises setting a first subset of bits of the first set of bits to indicate a minimum value of the first set of values, setting a second subset of bits of the first set of bits to indicate a maximum value of the first set of values, and setting a third subset of bits of the first set of bits to indicate a first portion of the data file separating the first metadata chunk and the second metadata chunk. The method also comprises receiving the search query specifying the particular data value, reading the first metadata chunk, and, in response to determining that the particular data value is equal to or greater than the minimum value and equal to or less than the maximum value, sending the first compressed data chunk to the one or more computing devices.