Columnar Table File Structures for Fast Entity Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current columnar storage formats in distributed database systems face issues with immutability, lack of awareness of application-level entities, and require tight coupling with centralized metadata stores, leading to inefficient updates and schema rigidity.
Innovation Solution
Implement a database system that stores table data in column files with entity chunks and embedded metadata, allowing soft updates and deletes, and uses file naming and placement conventions to eliminate the need for a centralized metadata repository.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If current columnar storage formats are used with immutable file structures, then data integrity is maintained, but update operations become costly and complex
Solution Approach 1:
The patent segments the columnar data structure into immutable data files and a separate mutable entity index. The entity index is divided into individual entity records that can be independently updated, deleted, or added without modifying the underlying data files. This segmentation allows updates to be performed by simply modifying index entries rather than rewriting entire data files, dramatically reducing update complexity while maintaining data integrity through the immutability of the data files.
2Ease of manufacture
If centralized metadata stores are used to manage table schema, then metadata management is centralized, but the system becomes rigid and prone to single points of failure
Solution Approach 1:
The patent extracts metadata management from a centralized metadata store and embeds it directly within the data files themselves. Each data file contains its own metadata headers that describe the schema and structure of the data within that file. This extraction eliminates the single point of failure associated with centralized metadata stores while maintaining ease of metadata management through self-describing data files that can be independently parsed and understood.
3Productivity
If columnar storage formats are designed for batch processing, then query performance is optimized, but per-entity update operations become inefficient
Solution Approach 1:
The patent introduces dynamic update capabilities to the traditionally static columnar storage format by implementing an entity index that can be dynamically modified. The entity index maintains mappings between entity identifiers and their corresponding data locations, allowing individual entities to be updated, deleted, or added without affecting the batch-optimized columnar data files. This dynamic index layer enables efficient per-entity operations while the underlying immutable data files maintain their batch processing performance characteristics.
Data Source
AI summary
A database system stores a table as a set of column files in a columnar format in a manner that improves the write performance of the table and avoids use of separate metadata repository. In embodiments, each column file groups values into entity chunks indexed by an entity index. Each chunk includes a live value index that determines which rows in chunk has live values. New values are written to the column file by appending an updated copy of the entity chunk. The entity index to refer to the newly written chunk as the latest version. This approach avoids expensive in-place updating of individual column values and allows the update to be performed much more quickly. In embodiments, the database system encodes metadata such as table schema information using file naming and placement conventions in the file store, so that a centralized metadata repository is not required.