Compressed Data Slabs With Position Metadata for Faster Database Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face limitations in processing speed due to hardware constraints, data storage methods, and restricted co-processing options, leading to inefficiencies in data management and query execution.
Innovation Solution
A database system architecture that employs parallelized data input, storage, and query processing, utilizing a parallelized data input sub-system, data store and process sub-system, and query and response sub-system, along with global dictionary compression techniques to optimize data handling and query execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in traditional uncompressed format, then storage simplicity is maintained, but storage space efficiency deteriorates
Solution Approach 1:
The patent segments data into fixed-size data slabs with structured headers containing metadata. Each slab is independently compressed and organized, allowing efficient storage and retrieval while maintaining overall system simplicity through standardized segmentation.
Solution Approach 2:
The patent applies compression algorithms that transform data from uncompressed to compressed format, changing the physical state of the data. This parameter change reduces storage space requirements while the structured slab format manages the increased complexity through standardized organization.
2Productivity
If parallel processing is implemented, then processing speed is improved, but system complexity increases
Solution Approach 1:
The patent divides data into separate slabs that can be processed in parallel by different computing resources. Each slab is independently addressable and can be compressed or queried simultaneously, enabling parallel processing while maintaining manageable system complexity through clear data partitioning.
Solution Approach 2:
The compressed data slabs are designed to be self-contained with embedded compression information and metadata, allowing them to be processed independently without requiring complex coordination between processing units. This self-service design simplifies parallel system architecture.
3Quantity of substance
If compression techniques are applied, then storage efficiency is improved, but query processing complexity increases
Solution Approach 1:
The patent pre-compresses data into standardized slabs with embedded compression metadata and headers before storage. This preliminary action organizes the compressed data structure in advance, so that query processing can directly utilize the pre-organized format without requiring complex decompression and reorganization operations during querying.
Solution Approach 2:
The patent introduces structured slab headers and compression information records as intermediary elements between the compressed data and query processing systems. These intermediaries contain metadata that enables efficient query operations on compressed data without requiring full decompression, thus reducing query processing complexity.
Data Source
AI summary
A computing entity is operable to a data set for storage that includes a plurality of uncompressed data slabs. A storage data set that includes a plurality of compressed data slabs is created based on the data set, and the storage data set is stored via a plurality of computing devices. Each compressed data slab of the plurality of compressed data slabs is generated from at least one corresponding uncompressed data slab of the plurality of uncompressed data slabs, and each compressed data slab is generated to include compressed data and compression information. The compressed data is generated based on determining non-null ones of the plurality of values of the at least one corresponding uncompressed data slab. The compression information is generated based on positions of the non-null ones of the plurality of values in accordance with a serialized value ordering.