Real-time Bitmap Index Generation with RLE Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High data rate streams, such as network traffic, pose challenges in generating efficient and accurate indexes in real-time due to increasing data volumes, leading to slower search and access times, making it difficult to index data promptly after recording.
Innovation Solution
A computer-implemented method using a bitmap index that applies run length encoding (RLE) schemes to sub-sequences of identical bit values, with multiple encoding types and a system that generates an index during data stream recording, allowing for efficient indexing and searching by indicating occurrences and non-occurrences of byte values, and utilizing intermediate arrays for precise location referencing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is recorded at high data rates, then the volume of data for recording increases, but the time required to generate an index for the data increases
Solution Approach 1:
The patent segments the index structure into multiple independent bitmaps, each representing a specific byte value (0-255). This allows parallel processing and independent generation of each bitmap, significantly reducing the overall index generation time while maintaining high data recording rates.
Solution Approach 2:
The patent performs preliminary actions by pre-allocating and pre-structuring the bitmap index framework before data recording begins. The index structure is prepared in advance with all necessary bitmaps and data structures ready, enabling immediate indexing as data arrives without setup delays.
2Quantity of substance
If the volume of data grows, then the time required to search and access the data increases
Solution Approach 1:
The patent divides the search space into 256 separate bitmaps (one for each byte value), allowing the search process to directly access specific bitmaps based on the search byte value. This segmentation enables constant-time search operations regardless of the total data volume, as each bitmap can be searched independently and efficiently.
Solution Approach 2:
The patent creates a compressed copy of the data pattern information in the bitmap index structure. Each bitmap serves as a compact representation of where specific byte values occur in the data stream, allowing rapid search without examining the actual data volume.
3Productivity
If a bitmap index is used to indicate occurrences of byte values, then the index size increases, but the search efficiency improves
Solution Approach 1:
The patent applies local quality optimization by using different run-length encoding strategies for different regions of the bitmap data. Specifically, it uses variable-length run-length encoding where the encoding parameters are adapted to the local characteristics of each bitmap section, achieving better compression ratios while maintaining search efficiency.
Solution Approach 2:
The patent changes the representation parameters of the bitmap index by implementing run-length encoding that transforms sequences of identical bit values into compressed representations. This parameter transformation reduces the index size while preserving the ability to efficiently search for byte value occurrences.
Data Source
AI summary
A computer implemented method of real-time index generation of data in a data stream during data stream recording to an archive, the method including: receiving a byte of the data stream; and adding an entry for the byte of the data stream to a bitmap index of bytes, the bitmap index indicating, for each possible byte value, a location in the archive of each of occurrences and non-occurrences of the byte value by a sequence of bit values in the bitmap index, wherein the bitmap index includes a count of a number of occurrences for each byte value, wherein a plurality of run length encoding (RLE) schemes are applied to sub-sequences of identical bit values for each byte value in the index, each RLE scheme having a different encoding type identified by an encoding type indicator and indicating a bit value and a number of bits in a sub-sequence, and wherein a pair of RLE encoded sub-sequences of indications of non-occurrence of a byte value implies a single indication of an occurrence of the byte value between the pair.