Distributed Object Store Metadata Query via Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current object stores lack query capabilities, making it inefficient to retrieve specific objects and metadata, as they typically require retrieving a superset of data and discarding unwanted items, which is resource-intensive for servers, networks, and applications.
Innovation Solution
Exposing object data and metadata to distributed processing systems like Hadoop, allowing for queries to be posed in a high-level declarative language, with secondary indexes defined based on access patterns, enabling efficient processing and retrieval of metadata records in formats compatible with distributed frameworks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If object stores retrieve a superset of objects and discard unwanted items, then query capabilities are provided, but resource consumption increases for servers, networks, and applications
Solution Approach 1:
The patent extracts only the necessary metadata fields from objects that satisfy the query predicates, rather than retrieving complete objects. The system identifies matching objects through predicates and extracts only the required metadata properties, eliminating the waste of transferring and processing unnecessary data.
Solution Approach 2:
The system performs partial action by retrieving only the subset of metadata that is needed for the query, not the complete object data. This allows the system to provide query capabilities while minimizing resource consumption by avoiding excessive data retrieval and processing.
2Measurement precision
If distributed processing systems process complete object data, then comprehensive analysis is achieved, but processing time and resource usage increase
Solution Approach 1:
The system extracts only the specific metadata fields required for the distributed processing analysis, rather than processing complete object data. This maintains analysis comprehensiveness for the required parameters while significantly reducing processing time and resource usage.
Solution Approach 2:
The system performs partial processing by focusing only on the metadata subset that is relevant to the query, enabling comprehensive analysis of the required parameters without the time penalty of processing all object data.
3Loss of information
If object stores provide full object retrieval, then data completeness is ensured, but network bandwidth consumption increases
Solution Approach 1:
The system extracts and transfers only the necessary metadata information that satisfies the query requirements, ensuring data completeness for the requested parameters while minimizing network bandwidth consumption by avoiding transmission of redundant data.
4Adaptability or versatility
If applications retrieve all objects for filtering, then query flexibility is achieved, but application processing overhead increases
Solution Approach 1:
The system extracts and provides only the metadata that matches the query predicates, maintaining query flexibility through predicate-based selection while reducing application processing overhead by eliminating the need for applications to filter large sets of complete objects.
Data Source
AI summary
A distributed object store can expose object metadata, in addition to object data, to distributed processing systems, such as Hadoop and Apache Spark. The distributed object store may acts as a Hadoop Compatible File System (HCFS), exposing object metadata as a collection of records that can be efficiently processed by MapReduce (MR) and other distributed processing frameworks. A distributed processing job can specify a metadata query to narrow the set of objects returned. Related methods are also described.