Database Query Routing Across Distributed Storage Centers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large scale unstructured database systems face challenges with distributed object storage, including slow access times, limited support for random access write or update operations, and complexities when integrating with dynamic schema databases like MongoDB, which require structured queries and data flattening, leading to loss of data fidelity.
Innovation Solution
The implementation of a system that supports read operations using a storage hierarchy including main memory and disk, with the option for distributed object storage, allowing for virtual 'collections' to be specified and queried analogous to document databases, and using storage configuration files to map object names to partitions for efficient querying across multiple data centers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If distributed object storage is used to store historical data, then storage cost is reduced and data durability is improved, but access speed decreases and random write/update operations are not supported
Solution Approach 1:
The system segments data storage into two distinct layers: hot data stored in online storage (main memory and disk) for fast access and random operations, and cold/historical data stored in distributed object storage for cost-effective durable storage. This segmentation allows each layer to optimize for its specific requirements without compromising the other.
Solution Approach 2:
The patent introduces an intermediary layer (online storage including main memory and disk) that mediates between the application layer and distributed object storage. This intermediary enables fast access and random write operations while allowing historical data to reside in the distributed object storage system, thus resolving the contradiction between speed and durability.
2Quantity of substance
If data is stored as blobs in distributed object storage, then storage volume increases and cost decreases, but query complexity increases and data fidelity is lost
Solution Approach 1:
The system performs preliminary actions by maintaining data in its original structured format in online storage before moving historical data to distributed object storage. Configuration files are prepared in advance with metadata information that enables efficient querying without requiring data flattening or restructuring, thus reducing query complexity while enabling massive storage volume in distributed object storage.
3Ease of operation
If structured queries are used to access distributed object storage, then data access becomes possible, but data fidelity is lost and flexibility decreases
Solution Approach 1:
The patent creates a virtual copy of the database structure through configuration files that map object names to partitions. This virtual structure allows applications to query distributed object storage using familiar database operations without actually restructuring the underlying data, thus maintaining data fidelity while improving ease of operation.
Data Source
AI summary
Systems, methods, and computer-readable media provide techniques for on-demand access to object data hosted in distributed object storage. A query is received for the distributed object storage database, wherein the distributed object storage database includes first object data stored in a first data center at a first geographical location, and second object data stored in a second data center at a second geographical location. It is determined that the query is for at least a portion of the first object data stored by the first data center. The query is transmitted to a first computing device in the first data center. Results are received from the first computing device for the query, and the results are provided in response to the received query.