Flash Memory Database Storage via Column Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Flash memory, which can only be written or erased a limited number of times before degradation, is suboptimal for storing databases due to the limitations on writing and rewriting.
Innovation Solution
A system and method that logically divides a database into portions, samples columns to identify compression strategies, and compresses data using dictionaries and metadata storage in flash memory, optimizing storage capacity and extending the lifespan of the memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in flash memory, then storage capacity is utilized, but the memory degrades due to limited write operations
Solution Approach 1:
The data file is logically divided into portions, and each portion is further divided into compression units. This segmentation allows the system to process and compress data in smaller chunks, reducing the frequency of full writes to flash memory and thereby extending its lifespan while maintaining storage capacity.
Solution Approach 2:
The system performs data compression and preprocessing in RAM before writing to flash memory. By preparing the data in advance with compression algorithms and organizing it into structured formats with metadata, the system reduces the volume of data that needs to be written to flash memory, thus reducing wear while maximizing storage utilization.
2Quantity of substance
If compression strategies are applied to data, then storage efficiency is improved, but processing complexity increases
Solution Approach 1:
The system applies different compression strategies to different columns based on their specific characteristics. By sampling columns to identify patterns and applying appropriate compression methods locally to each column rather than using a uniform approach, the system achieves high storage efficiency while keeping the processing complexity manageable through targeted rather than universal processing.
Solution Approach 2:
The system dynamically selects compression strategies based on data characteristics such as null value density, value ranges, and patterns. By changing compression parameters adaptively according to the specific properties of each column and compression unit, the system optimizes storage efficiency without requiring overly complex fixed algorithms.
3Stability of the object's composition
If data is compressed into fixed-size buffers, then storage organization is improved, but data retrieval complexity increases
Solution Approach 1:
The system pre-organizes compressed data into fixed-size buffers (e.g., 32KB) and pre-computes metadata including compression unit boundaries, column pointers, and data offsets during the write process. This preliminary organization allows for efficient data retrieval without requiring complex real-time calculations, as all structural information is prepared in advance and stored with the data.
Solution Approach 2:
The system introduces metadata as an intermediary layer between the compressed data buffers and the retrieval process. This metadata includes pointers to compression units, column information, and buffer locations, which simplifies data retrieval by providing a structured map that translates high-level data requests into specific buffer and offset locations without requiring complex analysis of the compressed data itself.
Data Source
AI summary
A system and method stores a database file into Flash memory or other write-constrained storage.