Distributed Segment Structure for Faster Database Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face limitations in processing speed due to hardware constraints, data storage methods, and restricted co-process options, which affect the efficiency of data storage and retrieval operations.
Innovation Solution
The implementation of a large-scale data processing network that includes a database system with parallelized sub-systems for data input, storage, retrieval, and query processing, utilizing a distributed architecture across multiple computing devices and storage clusters to enhance processing speed and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in a traditional centralized database system, then data storage is simplified, but processing speed is limited by hardware constraints
Solution Approach 1:
The patent divides the database system into multiple distributed computing devices, each handling specific data segments. This segmentation allows parallel processing across devices, overcoming the processing speed limitations of centralized systems while distributing the computational load to manage complexity.
Solution Approach 2:
The patent transitions from a single-dimension centralized storage model to a multi-dimensional distributed architecture across multiple computing devices and storage clusters. This dimensional expansion enables parallel processing paths and improves overall system throughput.
2Productivity
If data is stored using conventional storage methods, then storage implementation is straightforward, but retrieval efficiency is reduced
Solution Approach 1:
The storage system is segmented into multiple storage clusters distributed across computing devices. Data is partitioned and stored across these segments, enabling parallel retrieval operations that improve efficiency while the modular segment structure helps manage architectural complexity.
Solution Approach 2:
The patent introduces intermediary components including indexing structures and metadata systems that facilitate efficient data location and retrieval across the distributed storage architecture. These intermediaries abstract the complexity of distributed storage while improving retrieval speed.
3Speed
If co-process options are restricted, then system design is simpler, but execution speed is limited
Solution Approach 1:
The distributed computing devices are designed with multi-functionality, capable of performing both data storage and various co-processing tasks including query processing, data transformation, and analytics. This universality enables flexible co-processing options across the distributed system while maintaining execution speed.
4Productivity
If data is stored in a distributed architecture across multiple computing devices, then processing speed is improved, but system complexity increases
Solution Approach 1:
The distributed system is segmented into standardized computing devices and storage clusters with defined interfaces and protocols. This segmentation creates modular units that can be independently managed and scaled, improving processing throughput while the modular nature helps manage overall system complexity.
Solution Approach 2:
The patent employs homogeneous computing devices with standardized architectures and interfaces throughout the distributed system. This homogeneity simplifies system management, deployment, and scaling operations while maintaining the parallel processing capabilities needed for high throughput.
Data Source
AI summary
A computing system is operable to generating a set of segments of a segment group. Each segment of the set of segments includes a data and parity section that includes a corresponding set of sorted data slabs, a manifest section that includes metadata regarding the corresponding set of sorted data slabs, at least one index section that includes index data for the corresponding set of sorted data slabs, and a statistics section storing statistical information regarding the each segments. The set of segments of the segment group are stored across a plurality of computing devices of a storage cluster of the computing system.