Columnar Compression Metadata for Scalable Database Writes and Reads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Row-based compression schemes in large-scale cloud-based multitenant database systems are inefficient, consuming significant system resources and affecting scalability due to their suboptimal compression and decompression processes.
Innovation Solution
Implementing column-based compression techniques that group and compress data of the same type across columns, allowing for more efficient data storage and retrieval by reducing the workload on database servers and utilizing application servers for compression and decompression operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If row-based compression schemes are used in database systems, then data can be stored and retrieved, but system resources are significantly consumed and scalability is affected
Solution Approach 1:
The patent segments data storage by column instead of by row, creating separate compression streams for each column. This allows independent compression of each column's data using column-specific compression algorithms, reducing overall system resource consumption while improving storage efficiency
Solution Approach 2:
The patent changes the fundamental parameter of data organization from row-based to column-based structure. This parameter change enables the use of column-specific compression algorithms that are optimized for the data types in each column, significantly reducing CPU overhead and improving compression ratios
2Quantity of substance
If row-based compression is implemented, then data storage is achieved, but compression and decompression processes are suboptimal and affect system scalability
Solution Approach 1:
The patent divides the compression process into separate operations for each column, allowing parallel processing and optimization of compression algorithms for each column's specific data type, thereby improving overall compression efficiency
Solution Approach 2:
The patent introduces columnar data structures as an intermediary layer between the database and storage system. This intermediary enables more efficient compression by grouping similar data types together, allowing specialized compression algorithms to process each group optimally
3Use of energy by moving object
If column-based compression techniques are used, then computational and storage resources are reduced, but data organization and retrieval mechanisms must be changed
Solution Approach 1:
The patent segments data into columns with uniform data types, which simplifies the compression process for each column while reducing overall computational and storage resources. The segmentation creates manageable units that can be processed independently
Solution Approach 2:
The patent creates a universal columnar storage framework that can handle multiple data types through a standardized interface. This universal structure reduces complexity by providing a consistent method for organizing and retrieving data regardless of the specific data type
Data Source
AI summary
A request for performing a data storing operation directed to a database table that comprises a plurality of table columns is received. Columnar compression metadata is accessed to identify one or more table columns in the database table, each of the one or more table columns being designated to store compressed columnar values. The columnar compression metadata is used to apply one or more columnar compression methods to generate, from one or more uncompressed columnar values received with the request for the data storing operation, one or more compressed columnar values to be persisted in the one or more table columns in the database table. A database statement is executed to persist the one or more compressed columnar values in the one or more table columns in the database table.