Key-Value Datastore Record Segmentation for Low-Latency Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data storage techniques, such as relational databases, can lead to delays in data access operations, especially in large data storage environments, due to inefficient handling of data access and storage processes.
Innovation Solution
Implementing a scalable data storage and retrieval system using a 'not only SQL' (NoSQL) database or key-value datastore that employs hash-based search and retrieval methods, allowing data to be apportioned into records with unique record keys for fast access and storage of large volumes of data, including transaction data, across various formats like textual, numerical, and binary data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Stability of the object's composition
If traditional relational database techniques are used for data storage, then data can be efficiently organized with data orthogonalization, but data access operations become slower due to the need to access multiple tables and resolve foreign keys
Solution Approach 1:
The patent segments data into separate collections based on entity types (users, posts, comments, etc.) rather than organizing all data in a single relational structure. Each collection stores related data independently, eliminating the need to traverse multiple tables and foreign keys during data access operations.
Solution Approach 2:
The patent introduces document identifiers and reference fields as intermediaries that directly link related data across collections. Instead of using foreign key relationships across multiple tables, the system uses embedded document references that can be directly resolved, reducing the complexity of data access operations.
2Device complexity
If traditional relational database techniques are used for data storage, then data can be structured with normalized relationships, but read and write operations experience delays in large data storage environments
Solution Approach 1:
The patent divides the database into multiple independent collections, each storing a specific type of data (users, posts, comments, etc.). This segmentation reduces the complexity of data structures within each collection and eliminates the need for complex joins across multiple normalized tables, thereby reducing operation delays.
Solution Approach 2:
The patent employs dynamic data structures that can adapt to different data types and relationships without requiring rigid pre-defined schemas. This allows for more flexible and efficient data access patterns compared to traditional relational databases, particularly in large-scale environments where data relationships can be complex and varied.
3Reliability
If data is stored in a single large structure, then data relationships can be maintained, but data retrieval becomes slower and less scalable
Solution Approach 1:
The patent segments data into separate collections while maintaining relationships through embedded references and document identifiers. This approach preserves data relationship integrity through direct references rather than indirect foreign key relationships, while simultaneously improving retrieval efficiency by allowing direct access to specific collections without scanning entire large structures.
Solution Approach 2:
The patent adds a new dimension to data organization by using hierarchical document structures within collections. Related data is organized in nested document structures that maintain relationships through embedded references, allowing for efficient retrieval along multiple dimensional paths (by collection, by document ID, by referenced documents) rather than through a single flat structure.
Data Source
AI summary
Transaction data is written to one or more records in a datastore using key-value pairs. The record key for a record comprises a hash key and a range key, where the hash key corresponds to a particular transaction and the range key corresponds to a transaction type. The record key also comprises a counter to distinguish between different records storing data for the same transaction. A serialized data stream of transaction data may be apportioned into multiple data records and stored in a non-relational datastore. Each record for a transaction is individually readable, independently of the other records for the transaction. Accordingly, data records storing a large amount of transaction data for a transaction may be individual retrieved and presented at an access device, enabling a paginated view of the large amount of data with low latency in its retrieval.