Dataset Snapshot Compression for RAM Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In video distribution systems and similar computing environments, storing datasets in RAM becomes inefficient as the dataset size increases, exceeding RAM capacity and leading to high latency when accessing remote locations, which is exacerbated by bandwidth limitations and the need for costly redesign of access patterns.
Innovation Solution
A dataset dissemination system that generates snapshots of datasets, using a write state engine to create compressed records, delta files, and reverse delta files, allowing for efficient storage and updates in RAM, reducing latency by enabling consumers to store snapshots locally and apply delta files to maintain dataset integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If the dataset is stored in RAM to reduce access latency, then the response time to end-user requests is improved, but the dataset size increases over time and may exceed the storage capacity of RAM
Solution Approach 1:
The patent divides the large dataset into multiple snapshots, each representing the dataset at a specific point in time. Each snapshot is stored in RAM separately, allowing the system to load only the necessary snapshot into memory rather than attempting to load the entire growing dataset. This segmentation enables efficient use of limited RAM resources while maintaining fast access to the required data version.
2Quantity of substance
If the dataset is stored in a remote location to accommodate growing size, then the storage capacity is sufficient, but the access latency increases to unacceptable levels
Solution Approach 1:
The patent creates and maintains multiple copies of the dataset in different locations and formats. Specifically, it generates snapshots of the dataset and stores them in a file store, while also maintaining an in-memory representation of the current snapshot. This copying approach allows the system to have a complete copy available in RAM for fast access, while the original growing dataset can be stored remotely without impacting performance.
3Productivity
If the entire dataset is copied to RAM to enable fast read-only operations, then the operation speed is improved, but the bandwidth consumption and copy time increase due to bandwidth limitations
Solution Approach 1:
The patent applies partial action by copying only the necessary portion of the dataset to RAM at any given time. Instead of copying the entire growing dataset, it copies only the current snapshot that is needed for operations. This partial copying significantly reduces bandwidth consumption and copy time while still enabling fast read-only operations on the required data.
Data Source
Figure 1
Figure 2A~2B
Figure 3A~3C
AI summary
In various embodiments, a write state application generates a snapshot that includes one or more data values associated with a source dataset. In operation, the write state application performs one or more compression operations on the source dataset to generate a first compressed record. The write state application then serializes the first compressed record and a second compressed record to generate a first compressed record list. Finally, the write state application generates the snapshot based on the first compressed record list. When the data values are accessed from the first snapshot, the size of the snapshot is maintained. Advantageously, because the size of the snapshot is smaller than the size of the source dataset, some consumers that are unable to store the entire source dataset in random access memory (RAM) are able to store the entire snapshot in RAM.