Relational Database Compression Using ML-Based Data Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database data compression methods, such as the ZSTD algorithm, are inefficient for structured data like financial and transaction data due to the low occurrence of duplicate character strings, resulting in low processing efficiency.
Innovation Solution
Classify database data using machine learning or deep learning algorithms to identify associations between rows and columns, then transform and compress data within the same classification categories using entropy coding, enhancing the compression ratio by increasing the proportion of duplicate characters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If the LZ77 algorithm is used to compress database data, then duplicate character strings can be eliminated, but processing efficiency is low because database data contains few duplicate character strings
Solution Approach 1:
The patent changes the fundamental parameter being compressed from character strings to numerical values. By extracting numerical features from database records and compressing these numeric parameters instead of text, the system achieves high compression ratios for financial and transaction data that contain minimal character duplication.
Solution Approach 2:
The patent replaces the mechanical string-matching mechanism of LZ77 with a statistical approach using numerical feature extraction and probability-based encoding. This substitution enables the system to handle structured numerical data efficiently without relying on character pattern repetition.
2Loss of substance
If general data compression algorithms are used, then high compression ratio can be achieved for text data, but they are ineffective for structured numerical data
Solution Approach 1:
The patent applies local quality by tailoring the compression approach specifically to numerical data characteristics. Instead of using a universal text-based compression method, it extracts and compresses numerical features locally within each record, adapting the compression strategy to the specific data type and structure.
Solution Approach 2:
The system changes the data representation parameter from textual format to numerical format, enabling specialized compression techniques that exploit statistical properties of numerical data. This parameter transformation makes the compression algorithm adaptable to structured data while maintaining high compression ratios.
Data Source
AI summary
This application discloses a database data compression method and a storage device, which are applicable to lossless compression of a relational database. The method includes parsing a first data block to obtain m pieces of first data, where the first data is row/column data, and the first data block is obtained based on database data. Then, the method includes classifying the m pieces of first data according to a machine learning algorithm or a deep learning algorithm, to obtain n classification categories, and transforming first data in each classification category to obtain target data. The method further includes compressing the target data to obtain compressed data. In this application, data in different rows/columns is classified according to the machine/deep learning algorithm, and then the row/column data belonging to the same classification category is transformed and compressed.