Object Storage Query Mechanism for Metadata Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current object storage systems lack query capabilities, forcing applications to retrieve a superset of objects and discard non-matching ones, which is resource-intensive and inefficient for both servers and networks.
Innovation Solution
Implementing a query mechanism within the object storage system that allows clients to pose queries in a high-level declarative language, using a query parser, optimizer, executor, and formatter to evaluate and return only relevant objects based on metadata predicates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If applications retrieve a superset of objects and discard non-matching ones, then query flexibility is maintained, but server resource usage and network bandwidth consumption increase significantly
Solution Approach 1:
The patent extracts the query processing functionality from the application layer and integrates it directly into the object storage system. The query parser, optimizer, and executor are embedded within the storage system, allowing queries to be processed during the object retrieval operation itself. This extraction eliminates the need for applications to retrieve and filter large supersets of objects, significantly reducing server resource usage while maintaining query flexibility.
Solution Approach 2:
The storage system is enhanced with multi-functionality by incorporating query processing capabilities alongside traditional storage and retrieval operations. The query executor can filter objects based on metadata predicates during the normal retrieval process, making the storage system universally capable of both storing objects and performing complex queries without requiring separate retrieval-and-filter operations.
2Adaptability or versatility
If applications retrieve a superset of objects and discard non-matching ones, then comprehensive data access is ensured, but network bandwidth consumption increases
Solution Approach 1:
The filtering operation is extracted from the application layer and performed within the storage system's query executor. This allows the system to retrieve only the specific objects matching the query predicates, rather than retrieving a comprehensive superset and filtering client-side. The network bandwidth consumption is reduced while maintaining comprehensive data access for matching objects.
Solution Approach 2:
The query processing and filtering actions are performed preliminarily within the storage system before objects are transmitted over the network. The query executor evaluates metadata predicates and filters objects during the retrieval process, so that only relevant objects are sent to the application. This preliminary filtering action eliminates unnecessary network traffic while ensuring comprehensive access to all matching objects.
3Adaptability or versatility
If current object storage systems retrieve all objects for client-side filtering, then query capability is achieved, but system efficiency and scalability deteriorate
Solution Approach 1:
The patent introduces an intermediary query processing layer between the application and the stored objects. This intermediary includes a query parser that converts application queries into executable plans, and a query optimizer that generates efficient retrieval strategies. This intermediary enables the system to achieve full query capability while maintaining high efficiency by processing queries during the normal storage system operations rather than requiring inefficient client-side filtering of all objects.
4Productivity
If query processing is added to object storage systems, then query efficiency improves, but system complexity increases
Solution Approach 1:
The query processing functionality is segmented into distinct modular components: a query parser that converts queries into parse trees, a query optimizer that generates and evaluates candidate query plans, and a query executor that performs the actual retrieval and filtering. This segmentation allows each component to be independently implemented and optimized, improving query efficiency while managing system complexity through clear separation of concerns and modular architecture.
Data Source
AI summary
An object storage system having object metadata query capability. Clients can pose queries to the object system in a high-level, declarative language. A query is evaluated against stored object metadata, and only objects satisfying the query's predicates are returned. The object storage system can evaluate alternative logical and physical query plans using a cost model. Queries can be posed using familiar REST-based APIs.