Distributed Columnar Data Storage with Dynamic Buffering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in efficiently storing and retrieving large data sets across multiple storage devices, particularly in managing constraints on data exchange, encryption, and decryption.
Innovation Solution
An apparatus and method that involve a node device receiving instructions from a control device to persistently store a data set, which includes instantiating collection threads, data buffers, and aggregation threads. The data values are reorganized from a row-wise to a columnar organization within row groups, and these are stored in data buffers. The method also includes dynamically adjusting the quantity of data buffers based on storage space availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If data values are stored in row-wise organization, then storage simplicity is improved, but data retrieval efficiency deteriorates
Solution Approach 1:
The patent segments the data storage structure into columnar units, where data values are organized by column rather than by row. This segmentation allows independent access to specific columns without retrieving entire rows, thereby improving data retrieval efficiency while maintaining storage organization.
Solution Approach 2:
The patent transitions from traditional row-wise storage to columnar storage, effectively changing the dimension of data organization. Instead of accessing data horizontally across rows, the system accesses data vertically through columns, enabling more efficient queries that operate on specific columns without retrieving unnecessary data from other columns.
2Quantity of substance
If large data sets are distributed across multiple storage devices, then storage capacity is improved, but data exchange constraints worsen
Solution Approach 1:
The patent divides large data sets into columnar segments that can be independently stored and managed across multiple storage devices. This segmentation reduces the complexity of data exchange by allowing selective transmission of only the required column segments rather than entire data sets or rows, thereby easing network bandwidth constraints.
Solution Approach 2:
The patent extracts specific column data from distributed storage locations and consolidates them for processing. This extraction approach allows the system to retrieve only the necessary data portions from multiple storage devices, reducing the overall data exchange volume and mitigating network constraints.
3Reliability
If data is encrypted for security, then data protection is improved, but processing speed deteriorates
Solution Approach 1:
The patent applies encryption to data before storing it in columnar format across distributed storage devices. By performing the encryption action in advance during the data ingestion phase, the system ensures data protection without impacting the speed of subsequent data retrieval and processing operations, as the encrypted data can be directly accessed and decrypted only when necessary.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An apparatus includes a processor to: instantiate collection threads, data buffers of a queue, and aggregation threads: within each collection thread, assemble a row group from a subset of the multiple rows, reorganize the data values row-wise to columnar organization, and store the row group within a data buffer of the queue; operate the buffer queue as a FIFO buffer; within each aggregation thread, retrieve multiple row groups from multiple data buffers of the queue, assemble a data set part from the multiple row groups, transmit, to storage device(s) via a network, the data set part; and in response to each instance of retrieval of a row group from a data buffer of the buffer queue for use within an aggregation thread, analyze a level of availability of at least storage space within the node device to determine whether to dynamically adjust the quantity of data buffers of the buffer queue.