Segmented Identifier Array Compression for Fast In-Memory Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In computing systems, large amounts of memory are required to store sorted arrays of identifiers, leading to resident memory fullness and potential data eviction to disk storage, which increases execution time and reduces system performance.
Innovation Solution
The method involves compressing sorted arrays of identifiers by splitting them into segments, generating a balanced binary search tree with nodes pointing to compressed segments, and maintaining these in memory for efficient querying, while also caching frequently used segments in decompressed form.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If sorted arrays of identifiers are stored in physical or main memory, then fast access is achieved, but memory usage increases causing data eviction to disk storage
Solution Approach 1:
The sorted array of identifiers is divided into multiple segments, where each segment is independently compressed. This allows the system to store compressed segments in memory while maintaining efficient access to individual segments through the binary search tree structure, reducing overall memory usage while preserving fast access capabilities.
Solution Approach 2:
The patent applies compression algorithms to transform the sorted array of identifiers into a compressed representation. By changing the parameter of data representation from uncompressed to compressed form, the memory footprint is reduced by approximately 39%, allowing more data to remain in memory without eviction to disk storage.
2Quantity of substance
If compressed segments are stored in memory with binary search tree structure, then memory footprint is reduced, but access time may increase
Solution Approach 1:
A binary search tree is constructed in advance, with each node containing metadata about compressed segments. This preliminary structure allows the system to quickly navigate to the relevant compressed segment without decompressing the entire array, maintaining efficient access times despite the compressed storage format.
Solution Approach 2:
The binary search tree acts as an intermediary structure between the query and the compressed segments. Instead of directly searching through compressed data, the system uses the binary search tree to locate the appropriate segment, reducing access time while maintaining the memory space benefits of compression.
Data Source
AI summary
A method includes obtaining an array of sorted identifiers to be stored in a designated portion of a memory of a given computing system, determining a segment size for splitting elements of the array into a plurality of segments, splitting the array into the plurality of segments based at least in part on the determined segment size, and compressing the plurality of segments to create a plurality of compressed segments. The method also includes generating a balanced binary search tree comprising a plurality of nodes each identifying a range of elements of the array corresponding to a given one of the segments and comprising a pointer to a given compressed segment corresponding to the given segment. The method further includes maintaining the balanced binary search tree and the compressed segments in the designated portion of the memory, and processing queries to the array utilizing the balanced binary search tree.