Layered Index for Variable-Length Data Random Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for random access to variable-length data incur significant storage overhead and time due to the need for extensive indexing, which increases storage space requirements and access time.
Innovation Solution
A layered index structure is implemented, comprising index arrays, block offset arrays, and per-block size arrays, allowing for compressed random access by representing indices as delta values from a block base, thereby reducing storage overhead and enabling O(1) access time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If extensive indexing is used for random access to variable-length data, then access time is improved, but storage overhead increases
Solution Approach 1:
The index is divided into multiple layers with each layer managing a subset of data blocks. The first layer indexes larger blocks while subsequent layers handle smaller blocks within those ranges. This segmentation allows efficient random access without requiring a single extensive index structure, thereby reducing overall storage overhead while maintaining fast access times.
Solution Approach 2:
The indexing structure uses a nested approach where smaller index structures are contained within larger ones. Each layer of the index contains references to blocks that are further divided into smaller blocks by subsequent layers. This nesting allows the system to achieve fast random access through the outer layers while storing compressed index information in inner layers, balancing access speed with storage efficiency.
2Ease of operation
If traditional indexing methods are used for variable-length data, then random access capability is provided, but both storage space and access time increase
Solution Approach 1:
The indexing system dynamically adapts to the variable-length nature of the data by using delta values that adjust to the actual block sizes. The index structure can accommodate different block lengths without requiring fixed-size entries, allowing efficient random access while minimizing storage space through flexible, dynamic index entries that reflect the actual data layout.
Data Source
AI summary
Aspects of the present disclosure provide a computer-implemented method that includes providing a layered index to variable length data, the layered index comprising a plurality of layers. Each layer of the plurality of layers has an index array, a block offset array, and a per-block size array. The index array identifies a next level index of a plurality of indices or data. The indices represent a delta value from a first index of a block. The block offset array identifies a starting location of the index array. The per-block array identifies a shared integer size of a block of indices. The method further includes performing a random access read of the variable length data using the layered index.