Ordered Associative Array on Append-Only Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage technologies using append-only storage media do not support ordering between pairs, limiting features like range queries and listing keys in sorted order, which are essential for efficient data retrieval and management.
Innovation Solution
Implementing an ordered associative array of pairs on top of append-only storage media by using a segment file and an attribute index, with asynchronous indexing and a multi-part hashing scheme to maintain offset locations, allowing for efficient data organization and retrieval while supporting ordering and range queries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If prior technology for storing non-ordered associative arrays is used on append-only storage media, then storage simplicity is maintained, but ordering capability and range query support are lost
Solution Approach 1:
The patent divides the storage structure into multiple segments, where each segment contains a sorted array of key-value pairs. This segmentation allows the system to maintain ordering within each segment while keeping the overall structure manageable on append-only storage media. The segment file stores multiple segments sequentially, and the attribute index provides mapping from keys to segment locations.
Solution Approach 2:
The patent introduces a new dimension of organization by creating a two-level structure: segments organized sequentially in the segment file (one dimension) and keys sorted within each segment (another dimension). This multi-dimensional organization enables both ordering capability and efficient range queries without requiring complex random access operations.
2Reliability
If append-only storage media is used, then write simplicity and durability are improved, but random access and sorting operations become difficult
Solution Approach 1:
The patent performs preliminary sorting of key-value pairs before writing them to segments in the segment file. By pre-sorting data during the write operation, the system ensures that ranges are naturally ordered without requiring complex random access or reordering operations later. This preliminary action enables efficient range queries while maintaining the simplicity of append-only writes.
3Adaptability or versatility
If ordering is implemented on append-only storage, then range queries become possible, but write performance and latency may increase
Solution Approach 1:
The patent segments the data storage into multiple independent segments within the segment file, allowing parallel or batched writes to different segments. This segmentation reduces the latency impact of sorting and ordering operations by distributing the workload across multiple segments rather than requiring a single large sorted structure.
Solution Approach 2:
The patent maintains sorted order within each segment rather than requiring global sorting across the entire dataset. This partial ordering approach reduces write latency by performing sorting only within smaller segment boundaries, accepting that some ordering work is done excessively (within each segment) in exchange for reduced overall complexity and latency.
Data Source
AI summary
The described technology is generally directed towards efficiently organizing data in a single, non-distributed database as an ordered associated array of key-value pairs implemented on top of a storage medium that allows only data appends. A table segment in an append-only storage medium is accessed by table keys. The table keys are hashed into key values used to access information in an attribute index (a search tree) that contains offsets to the table segment entries. Hashing the table keys can result in a series of hash parts, including a primary hash part corresponding to a node in the attribute index, and as needed in the event of hash collisions, secondary hash parts that map to child nodes in the attribute index. The table keys are additionally organized in a sorted data structure which is stored within the table segment along with other keys.