Columnar Storage Writes Using Independent Column-Based Caches
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for storing columnar data in storage face inefficiencies due to large sequential writes, which are slow and not optimally handled by certain storage architectures, while smaller concurrent writes are faster but not adequately supported.
Innovation Solution
Implementing multiple smaller column-based caches (CBCs) for writing data, allowing independent flushing of full caches without waiting for others, enabling concurrent writes of smaller data blocks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is written to storage using a single large cache with sequential writes, then the write operation is simpler to manage, but write performance is slower
Solution Approach 1:
The patent divides a single large cache into multiple smaller column-based caches (CBCs), each dedicated to storing data for a specific column. This segmentation enables concurrent writes to different columns simultaneously, dramatically improving write performance. The format handler manages multiple CBCs independently, allowing parallel data processing and elimination of sequential write bottlenecks.
2Adaptability or versatility
If data is written as large sequential blocks, then fewer write operations are needed, but storage architectures like object storage cannot adequately handle them
Solution Approach 1:
The patent segments data into smaller 1 MB blocks that are well-suited for object storage and other modern storage architectures. By dividing large sequential writes into manageable concurrent writes of smaller blocks, the system achieves both compatibility with various storage systems and improved write throughput through parallel processing.
Solution Approach 2:
The system dynamically adjusts write operations by independently flushing each column-based cache when it becomes full, rather than waiting for all caches to fill. This dynamic approach allows flexible concurrency and optimizes write speed while maintaining compatibility with storage architectures that handle smaller, more frequent write operations better.
Data Source
AI summary
Techniques described herein relate to a method for storing data in columnar storage. The method includes obtaining a columnar storage write request associated with a file, wherein the file comprises rows and columns of file data; in response to obtaining the columnar storage write request: writing the file data to storage using column-based caches; generating file metadata based on the writing of the file data to the storage; and assigning a key to the file metadata; and storing the file metadata using a key-value service.