In-Memory Storage Compression With Zero Read Amplification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In-memory data storage systems face significant cost issues due to high memory resource requirements and computation costs associated with data format conversion and compression, leading to resource conflicts between the application layer and the underlying storage layer in large-scale parallel processing platforms.
Innovation Solution
A storage-aware memory controller that maps physical memory into a memory region and a storage region, applying fine-grained and coarse-grained error protection schemes, and uses an in-memory storage filesystem for compressing data into codeword-aligned blocks with compression-aware logical block addresses to reduce memory resource costs and eliminate CPU workload for compression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is stored in storage-oriented format (JSON, Parquet, ORC) in in-memory storage systems, then data can be shared and exchanged among different in-memory computation frameworks, but data format conversion is required when moving data across layers, resulting in significant computation cost
Solution Approach 1:
The patent applies preliminary action by pre-converting data to storage-oriented formats (JSON, Parquet, ORC) before storage in the in-memory storage layer, and pre-computing format conversion paths. This allows the data to be stored in a universally compatible format while minimizing runtime conversion costs by having conversion routines pre-prepared and optimized.
Solution Approach 2:
The patent introduces an intermediary conversion layer between the application layer and storage layer. This intermediary component handles format conversions efficiently by maintaining conversion routines and data format mappings, thereby reducing the computation burden on the main processing systems while enabling versatile data sharing across different frameworks.
2Quantity of substance
If data compression is applied to reduce memory capacity overhead of in-memory data storage, then memory resource cost is reduced, but compression operations lead to further computation costs
Solution Approach 1:
The patent applies local quality by implementing selective compression where only specific data blocks or pages that benefit from compression are compressed, while hot data or frequently accessed data remains uncompressed. This localized approach reduces overall memory capacity requirements while minimizing computation costs by avoiding unnecessary compression of data that doesn't benefit from it.
Solution Approach 2:
The patent implements partial compression action by compressing only a portion of the data storage capacity rather than applying compression uniformly to all data. This allows the system to achieve memory capacity reduction through compression of suitable data while avoiding the computation overhead of compressing all data, thereby balancing memory efficiency with computation cost.
3Stability of the object's composition
If in-memory data storage systems hold data in storage-oriented format rather than in-memory objects, then data can be persisted and shared, but data format conversion is required when moving data, resulting in computation resource confliction between application layer and storage layer
Solution Approach 1:
The patent applies segmentation by dividing the system into distinct layers (application layer, in-memory storage layer, persistent storage layer) with clearly defined responsibilities. The in-memory storage layer maintains data in storage-oriented formats for persistence, while the application layer handles format-specific processing. This segmentation reduces computation resource confliction by localizing format conversion tasks to specific layers and avoiding redundant conversions.
Data Source
AI summary
A storage aware memory controller and method for managing a physical storage system. A described controller includes: a system for mapping physical memory space into a memory region and a storage region; a system for applying different error protections schemes, in which a fine-grained memory fault tolerance scheme is applied to data in the memory region and a course-grained memory fault tolerance scheme is applied to data in the storage region; and an in-memory storage filesystem that compresses and stores individual pages of data in the storage region, wherein each page of data is compressed into a set of codewords that are codeword aligned such that no codeword shares compressed data from different pages, and wherein the in-memory storage filesystem stores a compression-aware logical block address (CA-LBA) for each page of data.