Locality Group File Storage for Distributed Data Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed data management systems face inefficiencies when using table data structures for large data storage, as they often require accessing a large number of files across the system, reducing overall efficiency.
Innovation Solution
The implementation of a table data structure that groups columns into locality groups, with each locality group stored as one or more files, allowing for efficient distribution and access by using locality group files, which can be compressed and managed to optimize storage and retrieval processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If table data structures are used to store large amounts of data in distributed systems, then data storage capacity is improved, but the number of files accessed during data operations increases, reducing system efficiency
Solution Approach 1:
The patent divides the table data structure into multiple partitions, where each partition contains a subset of the table's columns. This segmentation allows data to be distributed across multiple files or storage locations, enabling parallel access and reducing the number of files that need to be accessed during data operations, thereby maintaining high storage capacity while improving system efficiency
Solution Approach 2:
The patent introduces a new organizational dimension by grouping columns into locality groups based on access patterns and data relationships. This creates a hierarchical structure where data can be accessed at different granularities, allowing the system to optimize file access by selecting appropriate locality groups rather than accessing all files, thus resolving the contradiction between storage capacity and access efficiency
2Adaptability or versatility
If columns are distributed across many files in a table structure, then data organization flexibility is improved, but data access efficiency deteriorates due to excessive file access
Solution Approach 1:
The patent applies local quality by creating locality groups with different organization characteristics optimized for specific access patterns. Frequently accessed columns are grouped together in hot locality groups, while less frequently accessed columns are placed in cold locality groups. This allows the system to adapt data organization to local access requirements, improving both flexibility and access efficiency simultaneously
Solution Approach 2:
The patent implements dynamic data organization through locality groups that can be adjusted based on access patterns. The system can dynamically create, modify, or merge locality groups to optimize for changing access requirements, allowing the data structure to adapt over time rather than being static, thus maintaining flexibility while reducing access time
Data Source
AI summary
Each of a plurality of data items is stored in a table data structure. The table structure includes a plurality of columns. Each of the columns is associated with one of a plurality of locality groups. Each locality group is stored as one or more corresponding locality group files that include the data items in the columns associated with the respective locality group. In some embodiments, the columns of the table data structure may be grouped into groups of columns and each group of columns is associated with one of a plurality of locality groups. Each locality group is stored as one or more corresponding locality group files that include the data items in the group of columns associated with the respective locality group.