Virtual Files in Columnar Database for Transaction Isolation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face challenges in isolating changes from concurrent transactions while maintaining ACID properties and providing fast, scalable access to large numbers of data records, especially in columnar databases.
Innovation Solution
The implementation of virtual files in a columnar database, where each virtual file is stored in physical data storage and can be transiently cached in a persistence layer, allowing changes from one transaction to be isolated from others, with metadata managing the most current version and supporting ACID properties.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If virtual files are used to store data records in a columnar database, then data isolation among concurrent transactions is improved, but access speed and scalability may deteriorate due to the complexity of managing virtual files and persistence layers
Solution Approach 1:
The patent segments the database storage into virtual files that can be independently managed and cached. Each virtual file represents a separate data record or portion of data, allowing individual caching strategies and isolation of transactions. This segmentation enables the persistence layer to manage specific virtual files without affecting others, thus maintaining data isolation while improving access speed through selective caching.
Solution Approach 2:
The patent introduces a persistence layer as an intermediary between the columnar data store and physical data storage. This persistence layer manages virtual files and their caching, acting as a mediator that handles the complexity of data isolation and persistence. By placing this intermediary layer, the system achieves reliable data isolation while the persistence layer optimizes access patterns to maintain speed.
2Reliability
If virtual files are instantiated and managed with metadata for version control, then ACID properties are improved, but device complexity increases due to metadata management and persistence layer architecture
Solution Approach 1:
The patent creates virtual files that serve multiple functions: they store data records, provide transaction isolation, enable version control through metadata, and support caching in the persistence layer. This multi-functionality reduces the need for separate mechanisms for each ACID property, thereby improving reliability while managing complexity through a unified virtual file abstraction.
Solution Approach 2:
The patent uses metadata to maintain copies of version information and references to virtual files. Instead of duplicating entire data sets, the metadata stores references and version information that allow the system to track and manage multiple versions efficiently. This copying approach ensures ACID properties through version control while minimizing the complexity overhead compared to full data duplication.
3Speed
If portions of virtual files are cached transiently in a persistence layer, then access speed is improved, but loss of information increases due to potential cache inconsistency
Solution Approach 1:
The patent implements dynamic caching in the persistence layer where portions of virtual files are cached transiently based on access patterns and transaction needs. The cache is not static but adapts to changing requirements, allowing the system to improve access speed for frequently accessed data while maintaining the ability to invalidate or update cache entries to prevent information loss. This dynamic approach balances speed improvement with consistency maintenance.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A plurality of virtual files are instantiated in a columnar database. The columnar database includes a columnar data store that persists tables of data contained therein to the plurality of virtual files. Each virtual file is stored in physical data storage and a portion of the virtual file can be cached transiently in a persistence layer intermediate the columnar data store and the physical data storage. The virtual files each comprise a database object for storing data records. Operation are processed, in the columnar database, using the virtual files such that changes to virtual files resulting from the operations of one transaction are isolated from changes on the same virtual files resulting from the operations of other transactions. Related apparatus, systems, techniques and articles are also described.