Local Persistent Storage Compaction for Low-Latency Data Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High throughput systems face inefficiencies in data access and management due to the need to persist data before processing, leading to system latency and resource consumption, especially in near real-time processing of large volumes.
Innovation Solution
A data management system generates a compacted view of external storage units in a local key-value store using replicated logs, creating indexes based on key mappings, and stores these in local persistent storage to facilitate efficient data access without relying on external systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is persisted to external storage units before processing, then data reliability is ensured, but system latency increases and resource consumption increases
Solution Approach 1:
The system performs preliminary actions by creating local copies of data from external storage units before processing occurs. The compactor proactively maintains local persistent storage with replicated log data, so when processing needs to occur, the data is already available locally without requiring external access during the critical processing path.
Solution Approach 2:
The patent introduces an intermediary layer (local persistent storage with compacted view) between the external storage units and the processing system. This intermediary holds replicated data locally, allowing processing to occur without direct external storage access, thus reducing latency while maintaining data reliability through the intermediary buffer.
2Reliability
If data is accessed from external storage units, then data availability is ensured, but resource consumption increases
Solution Approach 1:
The system implements local quality by maintaining local copies of frequently accessed data in local persistent storage. The compactor creates a compacted view tailored to local processing needs, storing only the necessary replicated logs locally rather than accessing all external storage units, thus reducing resource consumption while ensuring data availability for processing.
Solution Approach 2:
The patent uses copying by creating replicated copies of external storage unit data in local persistent storage. The compactor generates local copies of the necessary data structures and logs, allowing processing to occur using these local copies rather than repeatedly accessing external storage, thereby reducing resource consumption while maintaining data availability.
3Speed
If a compacted view is created in local persistent storage, then data access speed improves, but device complexity increases
Solution Approach 1:
The patent applies segmentation by dividing the storage system into distinct components: external storage units, local persistent storage, and the compactor. The compacted view is segmented as a separate data structure that maps external storage offsets to local copies, allowing fast access without requiring the entire external storage system to be accessed, thus improving speed while managing complexity through modular segmentation.
4Productivity
If local persistent storage is used for data access, then processing throughput increases, but initial setup complexity increases
Solution Approach 1:
The compactor operates as a self-service component that automatically manages the creation and maintenance of the compacted view in local persistent storage. It autonomously replicates logs from external storage units, updates the compacted view as data changes, and manages local copies without requiring manual intervention, thus enabling high processing throughput while managing setup complexity through automation.
Data Source
AI summary
Various embodiments include systems, methods, and non-transitory computer-readable media for managing data using persistent storage. Consistent with these embodiments, a method includes receiving a write request that includes a value and a key; processing the write request, the processing of the write request including generating a message that includes the value; transmitting the message to an external system for storage of the value; generating an index based on a mapping of the key to an offset; and storing the value and the index in a local persistent storage.