Key-Value Store on Append-Only Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage technologies, such as DELL's Pravega, expose streams to external applications, requiring users to conform to specific semantics and limitations, which can be restrictive for data storage and retrieval, especially for non-ordered associative arrays of pairs and large objects.
Innovation Solution
Implementing a key-value store on top of an append-only storage medium that allows only data appends, using a non-distributed database with a segment file and attribute index, where keys map to offset locations, and employing a multi-part hashing scheme with asynchronous indexing to manage collisions efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If streaming data storage is used to provide flexibility and atomic durable appends, then data storage flexibility and reliability are improved, but users must conform to stream semantics and limitations which restricts data storage and retrieval operations
Solution Approach 1:
The patent introduces an intermediary layer (the key-value store system with segment files and attribute indexes) between the append-only storage medium and users. This intermediary translates high-level key-value operations into low-level append operations, shielding users from stream semantics while maintaining the benefits of append-only storage. The system handles complexity internally through components like the attribute index and segment file management.
2Productivity
If a non-distributed database with segment files and attribute indexes is implemented on append-only storage, then data retrieval efficiency and flexibility are improved, but system complexity increases
Solution Approach 1:
The patent segments the storage system into distinct components: segment files for data storage and attribute indexes for key-to-offset mapping. This segmentation allows independent optimization of each component and simplifies management by dividing the complex task of key-value storage into manageable parts with clear responsibilities.
Solution Approach 2:
The system performs preliminary actions by pre-computing and storing offset information in attribute indexes before data retrieval is needed. When data needs to be retrieved, the offset is already available in the index, enabling direct access without scanning the entire segment file. This preliminary indexing action significantly improves retrieval efficiency.
3Quantity of substance
If multi-part hashing with asynchronous indexing is used to manage collisions, then data storage capacity and collision handling are improved, but index update frequency and processing overhead increase
Solution Approach 1:
The patent employs periodic action through asynchronous indexing, where index updates are performed periodically or in batches rather than immediately with every data append. This allows the system to accumulate multiple data appends and process them together, reducing the frequency of index update operations and associated processing overhead while still maintaining data integrity.
Data Source
AI summary
The described technology is generally directed towards efficiently organizing data in a single, non-distributed database as an associated array of key-value pairs implemented on top of a storage medium that allows only data appends. A table segment in an append-only storage medium is accessed by table keys. The table keys are hashed into key values used to access information in an attribute index (a search tree) that contains offsets to the table segment entries. Hashing the table keys can result in a series of hash parts, including a primary hash part corresponding to a node in the attribute index, and as needed in the event of hash collisions, secondary hash parts that map to child nodes in the attribute index.