Sparse File Database Storage Model for Metadata Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional database storage models face performance issues and require substantial development effort, especially when managing large numbers of files in distributed clustered environments, leading to inefficiencies in storage and metadata management.
Innovation Solution
A database storage model utilizing sparse files with pre-allocated, fixed-sized segments at configurable offsets, allowing for efficient mapping and management of database objects within a collection file, reducing the need for individual files and simplifying disk space management through a catalog-based system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If individual files are used to represent each database object, then data organization and access efficiency are improved, but the number of files becomes unmanageably large (thousands to millions of files), leading to poor performance in metadata intensive operations
Solution Approach 1:
The patent merges multiple database object files into a single collection file. Instead of managing thousands of individual files for tables, indexes, and triggers, the system consolidates them into one file with internally managed segments, dramatically reducing the file count while maintaining organized access through segment-based structure.
Solution Approach 2:
The collection file is divided into fixed-size segments that can be independently managed and mapped to specific database objects. Each segment acts as a container for database object data, allowing the system to maintain granular control over data organization without requiring separate files for each object.
2Quantity of substance
If thousands to millions of individual files are created to represent database objects in a large MPP database, then comprehensive data storage is achieved, but concurrent creation and deletion of files becomes metadata intensive and performs poorly
Solution Approach 1:
By consolidating storage into a single collection file, the system eliminates the need for concurrent file creation and deletion operations. Database objects are created and dropped by allocating and deallocating segments within the existing file, transforming file-system intensive operations into simpler in-memory catalog updates.
Solution Approach 2:
The collection file is pre-established as a single container, eliminating the need for dynamic file creation during database operations. All database objects are accommodated within this pre-created file structure, with segments allocated as needed without triggering file system operations.
3Ease of manufacture
If traditional file system files are used to store each database object, then simple implementation is achieved, but performance is significantly lost or thrashing occurs in large database systems
Solution Approach 1:
The system implements a segmented storage structure within the collection file, dividing it into fixed-size segments that can be efficiently managed. This segmentation enables performance optimization through predictable I/O patterns and reduced metadata overhead, while maintaining implementation simplicity through a unified file interface.
Solution Approach 2:
The patent changes the storage model from variable-size individual files to fixed-size segments within a single file. This parameter change enables more efficient space utilization, reduced file system overhead, and improved performance through batch operations, while the configurable segment size allows optimization for different workload characteristics.
4Device complexity
If a collection file contains data for multiple database objects, then file management is simplified, but efficient access to specific objects requires effective mapping and indexing mechanisms
Solution Approach 1:
The patent introduces a catalog as an intermediary structure that maps database objects to their segment locations within the collection file. This catalog acts as an index, allowing the system to simplify file management by using a single file while efficiently locating specific objects through the catalog's mapping information.
Solution Approach 2:
The system pre-establishes a catalog structure that contains mapping information for all database objects to their segment locations. This preliminary organization enables efficient object retrieval without requiring complex search operations, as the catalog provides direct mapping from object identifiers to physical locations in the collection file.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Embodiments are provided herein for an efficient database storage model, which utilizes sparse file features to efficiently store and retrieve data. The embodiments provide database algorithms that utilize the file system abstraction layer to hide the complexity of managing disk space while providing the database a linear and contiguous logical address space for holding multiple database objects. An embodiment method includes pre-allocating, in a logical sparse file, a plurality of segments fixed in size and contiguous at fixed offsets. Upon receiving a command to write database objects to the segments, the database objects are mapped to the segments in a database catalog. The method further includes interfacing with a file system to initialize storage medium space for writing the data objects to the segments at the fixed offsets.